IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] Port docs to release branch (#18317)

Browse Source 
* [DOCS] Local distribution page improvements  (#18049)

* add slider with os specific libs

* doc review

* local distrib doc changes

* [DOCS] Added local distribution libraries path (#18191)

* add relative path to the table

* add another column

* new table format

* fix build issue

* fix tab name

* remove old table

* format fixes

* change font

* change path windows

* change tabset name

* add arm and 86_64 tables

* remove list dots

* [DOCS] Add FrontEnd API note (#18154)

* add note

* fix typo

* add advance cases note

* tf doc note

* wording change
This commit is contained in:
Tatiana Savina
2023-06-30 15:34:22 +02:00
committed by GitHub
parent bb1e353e58
commit f4709ffe8b
5 changed files with 98 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/MO_DG/prepare_model/convert_model/Convert_Model_From_ONNX.md
View File

@@ -7,6 +7,8 @@ Introduction to ONNX
`ONNX <https://github.com/onnx/onnx>`__ is a representation format for deep learning models that allows AI developers to easily transfer models between different frameworks. It is hugely popular among deep learning tools, like PyTorch, Caffe2, Apache MXNet, Microsoft Cognitive Toolkit, and many others.
.. note:: ONNX models are supported via FrontEnd API. You may skip conversion to IR and read models directly by OpenVINO runtime API. Refer to the :doc:`inference example <openvino_docs_OV_UG_Integrate_OV_with_your_application>` for more details. Using ``convert_model`` is still necessary in more complex cases, such as new custom inputs/outputs in model pruning, adding pre-processing, or using Python conversion extensions.
Converting an ONNX Model
########################

9
docs/MO_DG/prepare_model/convert_model/Convert_Model_From_Paddle.md
View File

@@ -2,6 +2,15 @@
@sphinxdirective
This page provides general instructions on how to convert a model from a PaddlePaddle format to the OpenVINO IR format using Model Optimizer. The instructions are different depending on PaddlePaddle model format.
.. note:: PaddlePaddle models are supported via FrontEnd API. You may skip conversion to IR and read models directly by OpenVINO runtime API. Refer to the :doc:`inference example <openvino_docs_OV_UG_Integrate_OV_with_your_application>` for more details. Using ``convert_model`` is still necessary in more complex cases, such as new custom inputs/outputs in model pruning, adding pre-processing, or using Python conversion extensions.
Converting PaddlePaddle Model Inference Format
##############################################
PaddlePaddle inference model includes ``.pdmodel`` (storing model structure) and ``.pdiparams`` (storing model weight). For how to export PaddlePaddle inference model, please refer to the `Exporting PaddlePaddle Inference Model <https://www.paddlepaddle.org.cn/documentation/docs/zh/develop/guides/beginner/model_save_load_cn.html>`__ Chinese guide.
To convert a PaddlePaddle model, use the ``mo`` script and specify the path to the input ``.pdmodel`` model file:
.. code-block:: sh

2
docs/MO_DG/prepare_model/convert_model/Convert_Model_From_TensorFlow.md
View File

@@ -4,6 +4,8 @@
This page provides general instructions on how to run model conversion from a TensorFlow format to the OpenVINO IR format. The instructions are different depending on whether your model was created with TensorFlow v1.X or TensorFlow v2.X.
.. note:: TensorFlow models are supported via :doc:`FrontEnd API <openvino_docs_MO_DG_TensorFlow_Frontend>`. You may skip conversion to IR and read models directly by OpenVINO runtime API. Refer to the :doc:`inference example <openvino_docs_OV_UG_Integrate_OV_with_your_application>` for more details. Using ``convert_model`` is still necessary in more complex cases, such as new custom inputs/outputs in model pruning, adding pre-processing, or using Python conversion extensions.
To use model conversion API, install OpenVINO Development Tools by following the :doc:`installation instructions <openvino_docs_install_guides_install_dev_tools>`.
Converting TensorFlow 1 Models

2
docs/MO_DG/prepare_model/convert_model/Convert_Model_From_TensorFlow_Lite.md
View File

@@ -8,7 +8,7 @@ To convert a TensorFlow Lite model, use the ``mo`` script and specify the path t
mo --input_model <INPUT_MODEL>.tflite
.. note:: TensorFlow Lite models are supported via FrontEnd API. You may skip conversion to IR and read models directly by OpenVINO runtime API.
.. note:: TensorFlow Lite models are supported via FrontEnd API. You may skip conversion to IR and read models directly by OpenVINO runtime API. Refer to the :doc:`inference example <openvino_docs_OV_UG_Integrate_OV_with_your_application>` for more details. Using ``convert_model`` is still necessary in more complex cases, such as new custom inputs/outputs in model pruning, adding pre-processing, or using Python conversion extensions.
Supported TensorFlow Lite Layers
###################################

149
docs/OV_Runtime_UG/deployment/local-distribution.md
View File

@@ -2,22 +2,22 @@
@sphinxdirective
With a local distribution, each C or C++ application/installer will have its own copies of OpenVINO Runtime binaries. However, OpenVINO has a scalable plugin-based architecture, which means that some components can be loaded in runtime only when they are really needed. Therefore, it is important to understand which minimal set of libraries is really needed to deploy the application. This guide helps you to achieve that goal.
With local distribution, each C or C++ application/installer has its own copies of OpenVINO Runtime binaries. However, OpenVINO has a scalable plugin-based architecture, which means that some components can be loaded in runtime only when they are really needed. This guide helps you understand what minimal set of libraries is required to deploy the application.
Local distribution is also appropriate for OpenVINO binaries built from sources using `Build instructions <https://github.com/openvinotoolkit/openvino/wiki#how-to-build>`__,
but the guide below supposes OpenVINO Runtime is built dynamically. For case of `Static OpenVINO Runtime <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/static_libaries.md>`__ select the required OpenVINO capabilities on CMake configuration stage using `CMake Options for Custom Compilation <https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/cmake_options_for_custom_comiplation.md>`__, the build and link the OpenVINO components into the final application.
.. note::
The steps below are operating system independent and refer to a library file name without any prefixes (like ``lib`` on Unix systems) or suffixes (like ``.dll`` on Windows OS). Do not put ``.lib`` files on Windows OS to the distribution, because such files are needed only on a linker stage.
The steps below are independent of the operating system and refer to the library file name without any prefixes (like ``lib`` on Unix systems) or suffixes (like ``.dll`` on Windows OS). Do not put ``.lib`` files on Windows OS to the distribution because such files are needed only at a linker stage.
Library Requirements for C++ and C Languages
############################################
Independent on the language used to write the application, the ``openvino`` library must always be put to the final distribution, since it's a core library which orchestrates with all the inference and frontend plugins. In Intel® Distribution of OpenVINO™ toolkit, ``openvino`` depends on the TBB libraries which are used by OpenVINO Runtime to optimally saturate the devices with computations, so it must be put to the distribution package.
Regardless of the programming language of an application, the ``openvino`` library must always be included in its final distribution. This core library manages all inference and frontend plugins. The ``openvino`` library depends on the TBB libraries which are used by OpenVINO Runtime to optimally saturate devices with computations.
If your application is written with C language, you need to put the ``openvino_c`` library additionally.
If your application is in C language, you need to additionally include the ``openvino_c`` library.
The ``plugins.xml`` file with information about inference devices must also be taken as a support file for ``openvino``.
@@ -34,76 +34,88 @@ Libraries for Compute Devices
For each inference device, OpenVINO Runtime has its own plugin library:
- ``openvino_intel_cpu_plugin`` for :doc:`Intel® CPU devices <openvino_docs_OV_UG_supported_plugins_CPU>`.
- ``openvino_intel_gpu_plugin`` for :doc:`Intel® GPU devices <openvino_docs_OV_UG_supported_plugins_GPU>`.
- ``openvino_intel_gna_plugin`` for :doc:`Intel® GNA devices <openvino_docs_OV_UG_supported_plugins_GNA>`.
- ``openvino_arm_cpu_plugin`` for :doc:`ARM CPU devices <openvino_docs_OV_UG_supported_plugins_CPU>`.
- ``openvino_intel_cpu_plugin`` for :doc:`Intel® CPU devices <openvino_docs_OV_UG_supported_plugins_CPU>`
- ``openvino_intel_gpu_plugin`` for :doc:`Intel® GPU devices <openvino_docs_OV_UG_supported_plugins_GPU>`
- ``openvino_intel_gna_plugin`` for :doc:`Intel® GNA devices <openvino_docs_OV_UG_supported_plugins_GNA>`
- ``openvino_arm_cpu_plugin`` for :doc:`ARM CPU devices <openvino_docs_OV_UG_supported_plugins_CPU>`
Depending on what devices are used in the app, the appropriate libraries need to be put to the distribution package.
Depending on which devices are used in the app, the corresponding libraries should be included in the distribution package.
As it is shown on the picture above, some plugin libraries may have OS-specific dependencies which are either backend libraries or additional supports files with firmware, etc. Refer to the table below for details:
As shown in the picture above, some plugin libraries may have OS-specific dependencies which are either backend libraries or additional supports files with firmware, etc. Refer to the table below for details:
.. dropdown:: Windows OS:
.. tab-set::
.. list-table::
:header-rows: 1
.. tab-item:: Windows
* - Device
- Dependency
* - CPU
- ``-``
* - GPU
- ``OpenCL.dll``, ``cache.json``
* - GNA
- ``gna.dll``
* - Arm® CPU
- ``-``
+--------------+-------------------------+-------------------------------------------------------+
| Device | Dependency | Location |
+==============+=========================+=======================================================+
| CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
| GPU | | OpenCL.dll | | ``C:\Windows\System32\opencl.dll`` |
| | | cache.json | | ``.\runtime\bin\intel64\Release\cache.json`` or |
| | | | ``.\runtime\bin\intel64\Debug\cache.json`` |
+--------------+-------------------------+-------------------------------------------------------+
| GNA | gna.dll | | ``.\runtime\bin\intel64\Release\gna.dll`` or |
| | | | ``.\runtime\bin\intel64\Debug\gna.dll`` |
+--------------+-------------------------+-------------------------------------------------------+
| Arm® CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
.. tab-item:: Linux arm64
+--------------+-------------------------+-------------------------------------------------------+
| Device | Dependency | Location |
+==============+=========================+=======================================================+
| Arm® CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
.. tab-item:: Linux x86_64
+--------------+-------------------------+-------------------------------------------------------+
| Device | Dependency | Location |
+==============+=========================+=======================================================+
| CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
| GPU | | libOpenCL.so | | ``/usr/lib/x86_64-linux-gnu/libOpenCL.so.1`` |
| | | cache.json | | ``./runtime/lib/intel64/cache.json`` |
+--------------+-------------------------+-------------------------------------------------------+
| GNA | libgna.so | ``./runtime/lib/intel64/libgna.so.3`` |
+--------------+-------------------------+-------------------------------------------------------+
.. dropdown:: Linux OS:
.. tab-item:: macOS arm64
.. list-table::
:header-rows: 1
+--------------+-------------------------+-------------------------------------------------------+
| Device | Dependency | Location |
+==============+=========================+=======================================================+
| Arm® CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
* - Device
- Dependency
* - CPU
- ``-``
* - GPU
- ``libOpenCL.so``, ``cache.json``
* - GNA
- ``gna.dll``
* - Arm® CPU
- ``-``
.. tab-item:: macOS x86_64
+--------------+-------------------------+-------------------------------------------------------+
| Device | Dependency | Location |
+==============+=========================+=======================================================+
| CPU | — | — |
+--------------+-------------------------+-------------------------------------------------------+
.. dropdown:: MacOS:
.. list-table::
:header-rows: 1
* - Device
- Dependency
* - CPU
- ``-``
* - Arm® CPU
- ``-``
Libraries for Execution Modes
+++++++++++++++++++++++++++++
The ``HETERO``, ``MULTI``, ``BATCH`` and ``AUTO`` execution modes can also be used explicitly or implicitly by the application. Use the following recommendation scheme to decide whether to put the appropriate libraries to the distribution package:
The ``HETERO``, ``MULTI``, ``BATCH`` and ``AUTO`` execution modes can also be used by the application explicitly or implicitly. Use the following recommendation scheme to decide whether to add the appropriate libraries to the distribution package:
- If :doc:`AUTO <openvino_docs_OV_UG_supported_plugins_AUTO>` is used explicitly in the application or `ov::Core::compile_model <classov_1_1Core.html#doxid-classov-1-1-core-1a46555f0803e8c29524626be08e7f5c5a>`__ is used without specifying a device, put ``openvino_auto_plugin`` to the distribution.
.. note::
Automatic Device Selection relies on :doc:`[inference device plugins <openvino_docs_OV_UG_Working_with_devices>`. If you are not sure about what inference devices are available on target system, put all the inference plugin libraries to the distribution. If `ov::device::priorities <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1gae88af90a18871677f39739cb0ef0101e>`__ is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only.
Automatic Device Selection relies on :doc:`inference device plugins <openvino_docs_OV_UG_Working_with_devices>`. If you are not sure which inference devices are available on the target system, put all inference plugin libraries in the distribution. If `ov::device::priorities <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1gae88af90a18871677f39739cb0ef0101e>`__ is used for `AUTO` to specify a limited device list, grab the corresponding device plugins only.
- If :doc:`MULTI <openvino_docs_OV_UG_Running_on_multiple_devices>` is used explicitly, put ``openvino_auto_plugin`` to the distribution.
- If :doc:`HETERO <openvino_docs_OV_UG_Hetero_execution>` is either used explicitly or `ov::hint::performance_mode <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1ga2691fe27acc8aa1d1700ad40b6da3ba2>`__ is used with GPU, put ``openvino_hetero_plugin`` to the distribution.
- If :doc:`BATCH <openvino_docs_OV_UG_Automatic_Batching>` is either used explicitly or ``ov::hint::performance_mode`` is used with GPU, put ``openvino_batch_plugin`` to the distribution.
- If :doc:`MULTI <openvino_docs_OV_UG_Running_on_multiple_devices>` is used explicitly, put ``openvino_auto_plugin`` in the distribution.
- If :doc:`HETERO <openvino_docs_OV_UG_Hetero_execution>` is either used explicitly or `ov::hint::performance_mode <groupov_runtime_cpp_prop_api.html#doxid-group-ov-runtime-cpp-prop-api-1ga2691fe27acc8aa1d1700ad40b6da3ba2>`__ is used with GPU, put ``openvino_hetero_plugin`` in the distribution.
- If :doc:`BATCH <openvino_docs_OV_UG_Automatic_Batching>` is either used explicitly or ``ov::hint::performance_mode`` is used with GPU, put ``openvino_batch_plugin`` in the distribution.
Frontend Libraries for Reading Models
+++++++++++++++++++++++++++++++++++++
@@ -111,16 +123,16 @@ Frontend Libraries for Reading Models
OpenVINO Runtime uses frontend libraries dynamically to read models in different formats:
- ``openvino_ir_frontend`` is used to read OpenVINO IR.
- ``openvino_tensorflow_frontend`` is used to read TensorFlow file format.
- ``openvino_tensorflow_lite_frontend`` is used to read TensorFlow Lite file format.
- ``openvino_onnx_frontend`` is used to read ONNX file format.
- ``openvino_paddle_frontend`` is used to read Paddle file format.
- ``openvino_tensorflow_frontend`` is used to read the TensorFlow file format.
- ``openvino_tensorflow_lite_frontend`` is used to read the TensorFlow Lite file format.
- ``openvino_onnx_frontend`` is used to read the ONNX file format.
- ``openvino_paddle_frontend`` is used to read the Paddle file format.
Depending on the model format types that are used in the application in `ov::Core::read_model <classov_1_1Core.html#doxid-classov-1-1-core-1ae0576a95f841c3a6f5e46e4802716981>`__, pick up the appropriate libraries.
Depending on the model format types that are used in the application in `ov::Core::read_model <classov_1_1Core.html#doxid-classov-1-1-core-1ae0576a95f841c3a6f5e46e4802716981>`__, select the appropriate libraries.
.. note::
To optimize the size of final distribution package, you are recommended to convert models to OpenVINO IR by using :doc:`model conversion API <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`. This way you don't have to keep TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, and other frontend libraries in the distribution package.
To optimize the size of the final distribution package, it is recommended to convert models to OpenVINO IR by using :doc:`model conversion API <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`. This way you do not have to keep TensorFlow, TensorFlow Lite, ONNX, PaddlePaddle, and other frontend libraries in the distribution package.
(Legacy) Preprocessing via G-API
++++++++++++++++++++++++++++++++
@@ -136,27 +148,34 @@ Examples
**CPU + OpenVINO IR in C application**
In this example, the application is written in C language, performs inference on CPU, and reads models stored as the OpenVINO IR format. The following libraries are used:
- The ``openvino_c`` library is a main dependency of the application. It links against this library.
In this example, the application is written in C, performs inference on CPU, and reads models stored in the OpenVINO IR format.
The following libraries are used: ``openvino_c``, ``openvino``, ``openvino_intel_cpu_plugin``, and ``openvino_ir_frontend``.
- The ``openvino_c`` library is a main dependency of the application. The app links against this library.
- The ``openvino`` library is used as a private dependency for ``openvino_c`` and is also used in the deployment.
- ``openvino_intel_cpu_plugin`` is used for inference.
- ``openvino_ir_frontend`` is used to read source models.
**MULTI execution on GPU and CPU in `tput` mode**
In this example, the application is written in C++, performs inference :doc:`simultaneously on GPU and CPU devices <openvino_docs_OV_UG_Running_on_multiple_devices>` with the `ov::hint::PerformanceMode::THROUGHPUT <enumov_1_1hint_1_1PerformanceMode.html#doxid-group-ov-runtime-cpp-prop-api-1gga032aa530efa40760b79af14913d48d73a50f9b1f40c078d242af7ec323ace44b3>`__ property set, and reads models stored in the ONNX format. The following libraries are used:
In this example, the application is written in C++, performs inference :doc:`simultaneously on GPU and CPU devices <openvino_docs_OV_UG_Running_on_multiple_devices>` with the `ov::hint::PerformanceMode::THROUGHPUT <enumov_1_1hint_1_1PerformanceMode.html#doxid-group-ov-runtime-cpp-prop-api-1gga032aa530efa40760b79af14913d48d73a50f9b1f40c078d242af7ec323ace44b3>`__ property set, and reads models stored in the ONNX format.
- The ``openvino`` library is a main dependency of the application. It links against this library.
The following libraries are used: ``openvino``, ``openvino_intel_gpu_plugin``, ``openvino_intel_cpu_plugin``, ``openvino_auto_plugin``, ``openvino_auto_batch_plugin``, and ``openvino_onnx_frontend``.
- The ``openvino`` library is a main dependency of the application. The app links against this library.
- ``openvino_intel_gpu_plugin`` and ``openvino_intel_cpu_plugin`` are used for inference.
- ``openvino_auto_plugin`` is used for Multi-Device Execution.
- ``openvino_auto_batch_plugin`` can be also put to the distribution to improve the saturation of :doc:`Intel® GPU <openvino_docs_OV_UG_supported_plugins_GPU>` device. If there is no such plugin, :doc:`Automatic Batching <openvino_docs_OV_UG_Automatic_Batching>` is turned off.
- ``openvino_auto_batch_plugin`` can be also put in the distribution to improve the saturation of :doc:`Intel® GPU <openvino_docs_OV_UG_supported_plugins_GPU>` device. If there is no such plugin, :doc:`Automatic Batching <openvino_docs_OV_UG_Automatic_Batching>` is turned off.
- ``openvino_onnx_frontend`` is used to read source models.
**Auto-Device Selection between GPU and CPU**
In this example, the application is written in C++, performs inference with the :doc:`Automatic Device Selection <openvino_docs_OV_UG_supported_plugins_AUTO>` mode, limiting device list to GPU and CPU, and reads models :doc:`created using C++ code <openvino_docs_OV_UG_Model_Representation>`. The following libraries are used:
In this example, the application is written in C++, performs inference with the :doc:`Automatic Device Selection <openvino_docs_OV_UG_supported_plugins_AUTO>` mode, limiting device list to GPU and CPU, and reads models :doc:`created using C++ code <openvino_docs_OV_UG_Model_Representation>`.
- The ``openvino`` library is a main dependency of the application. It links against this library.
The following libraries are used: ``openvino``, ``openvino_auto_plugin``, ``openvino_intel_gpu_plugin``, and ``openvino_intel_cpu_plugin``.
- The ``openvino`` library is a main dependency of the application. The app links against this library.
- ``openvino_auto_plugin`` is used to enable Automatic Device Selection.
- ``openvino_intel_gpu_plugin`` and ``openvino_intel_cpu_plugin`` are used for inference. AUTO selects between CPU and GPU devices according to their physical existence on the deployed machine.
- No frontend library is needed because ``ov::Model`` is created in code.