IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[DOCS] PT and TF tutorial (#21815)

Browse Source 
* add docs initial draft

* fix tabs

* move to learn ov section

* split the doc

* change model prep structure

* change content

* restructure conversion docs

* change docs order

* fix format issues

* fix format

* change blocks

* fix links

* fix build

* fix link

* change tutorial

* add links and fix wording

* change header

* fix name
This commit is contained in:
Tatiana Savina 2023-12-21 14:57:41 +01:00 committed by GitHub
parent 3546b0e15d
commit 243602929f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 403 additions and 118 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/articles_en/documentation/openvino_legacy_features.rst
View File

@ -82,7 +82,7 @@ offering.
than the remaining ones, their support has been discontinued. Converting them to the
ONNX format is a possible way of retaining them in the OpenVINO-based pipeline.
| :doc:`See the previous conversion instructions <mxnet_caffe_kaldi>`
| :doc:`See the currently supported frameworks <Supported_Model_Formats>`
| :doc:`See the currently supported frameworks <openvino_docs_model_processing_introduction>`
| **Post-training Optimization Tool (POT)**

2
docs/articles_en/documentation/openvino_legacy_features/mo_ovc_transition/legacy_conversion_api/supported_model_formats.rst
View File

@ -7,7 +7,7 @@
The code described here has been **deprecated!** Do not use it to avoid working with a legacy solution. It will be kept for some time to ensure backwards compatibility, but **you should not use** it in contemporary applications.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Supported Model Formats <Supported_Model_Formats>` article.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Supported Model Formats <openvino_docs_model_processing_introduction>` article.
.. toctree::
:maxdepth: 1

2
docs/articles_en/documentation/openvino_legacy_features/mo_ovc_transition/legacy_conversion_api/supported_model_formats/convert_model_tutorials/Convert_FaceNet_From_Tensorflow.rst
View File

@ -12,7 +12,7 @@ Converting TensorFlow FaceNet Models
The code described here has been **deprecated!** Do not use it to avoid working with a legacy solution. It will be kept for some time to ensure backwards compatibility, but **you should not use** it in contemporary applications.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Supported Model Formats <Supported_Model_Formats>` article.
This guide describes a deprecated conversion method. The guide on the new and recommended method can be found in the :doc:`Supported Model Formats <openvino_docs_model_processing_introduction>` article.
`Public pre-trained FaceNet models <https://github.com/davidsandberg/facenet#pre-trained-models>`__ contain both training
and inference part of graph. Switch between this two states is manageable with placeholder value.

4
docs/articles_en/learn_openvino.rst
View File

@ -17,6 +17,7 @@ Learn OpenVINO
Interactive Tutorials (Python) <tutorials>
Sample Applications (Python & C++) <openvino_docs_OV_UG_Samples_Overview>
Generative AI Optimization and Deployment <gen_ai_guide>
Import TensorFlow and PyTorch Models <openvino_docs_model_processing_introduction_draft>
This section will help you get a hands-on experience with OpenVINO even if you are just starting
@ -32,3 +33,6 @@ as well as an experienced user.
| :doc:`Optimize and Deploy Generative AI Models <gen_ai_guide>`
| Detailed information on how OpenVINO accelerates Generative AI use cases and what models it supports. This tutorial provides instructions for running Generative AI models using Hugging Face Optimum Intel and Native OpenVINO APIs.
| :doc:`Import TensorFlow and PyTorch Models <openvino_docs_model_processing_introduction_draft>`
| Learn about different import methods for TensorFlow and PyTorch models.

30
docs/articles_en/openvino_workflow.rst
View File

@ -26,7 +26,7 @@ OpenVINO offers multiple workflows, depending on the use case and personal or pr
This section will give you a detailed view of how you can go from preparing your model,
through optimizing it, to executing inference, and deploying your solution.
Once you obtain a model in one of the :doc:`supported model formats <Supported_Model_Formats>`,
Once you obtain a model in one of the :doc:`supported model formats <openvino_docs_model_processing_introduction>`,
you can decide how to proceed:
.. tab-set::
@ -48,6 +48,34 @@ you can decide how to proceed:
:align: center
:alt: OpenVINO workflow diagram for performance
OpenVINO uses the following functions for reading, converting, and saving models:
.. tab-set::
.. tab-item:: read_model
* Creates an ov.Model from a file.
* Supported file formats: OpenVINO IR, ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite. PyTorch files are not directly supported.
* OpenVINO files are read directly while other formats are converted automatically.
.. tab-item:: compile_model
* Creates an ov.CompiledModel from a file or ov.Model object.
* Supported file formats: OpenVINO IR, ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite. PyTorch files are not directly supported.
* OpenVINO files are read directly while other formats are converted automatically.
.. tab-item:: convert_model
* Creates an ov.Model from a file or Python memory object.
* Supported file formats: ONNX, PaddlePaddle, TensorFlow and TensorFlow Lite.
* Supported framework objects: PaddlePaddle, TensorFlow and PyTorch.
* This method is only available in the Python API.
.. tab-item:: save_model
* Saves an ov.Model to OpenVINO IR format.
* Compresses weights to FP16 by default.
* This method is only available in the Python API.
| :doc:`Model Preparation <openvino_docs_model_processing_introduction>`

62
docs/articles_en/openvino_workflow/model_preparation.rst
View File

@ -12,36 +12,33 @@ Model Preparation
:maxdepth: 1
:hidden:
Convert to OpenVINO Model <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>
Conversion Parameters <openvino_docs_OV_Converter_UG_Conversion_Options>
Setting Input Shapes <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Converting_Model>
Convert from PyTorch <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch>
Convert from TensorFlow <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow>
Convert from ONNX <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_ONNX>
Convert from TensorFlow_Lite <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite>
Convert from PaddlePaddle <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_Paddle>
Supported_Model_Formats
PyVision preprocessing <pytorch_vision>
You can obtain a model in one of :doc:`supported formats <Supported_Model_Formats>`
You can obtain a model in one of supported formats, **PyTorch, TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle**,
in many ways. The easiest one is to download it from an online database,
such as `TensorFlow Hub <https://tfhub.dev/>`__, `Hugging Face <https://huggingface.co/>`__,
and `Torchvision models <https://pytorch.org/hub/>`__. Now you have two options:
* Skip model conversion and run inference directly from the source format. Conversion
will still be performed but it will happen automatically and "under the hood."
* Skip model conversion and `run inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>`__ directly from the **TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle** source format. Conversion
will still be performed but it will happen automatically and "under the hood".
This option, while convenient, offers lower performance and stability, as well as
fewer optimization options.
* Explicitly convert the model to :doc:`OpenVINO IR <openvino_ir>`.
* Explicitly :doc:`convert the model to OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`.
This approach offers the best possible results and is the recommended one,
especially for for production-ready solutions. Explicit conversion can be done in two ways:
especially for production-ready solutions. Consider storing your model in this format to minimize first-inference latency,
perform model optimizations, and save space on your drive, in some cases. Explicit conversion can be done in two ways:
* the Python API functions (``openvino.convert_model`` and ``openvino.save_model``)
* the ``ovc`` command line tool.
Once saved as :doc:`OpenVINO IR <openvino_ir>` (a set of ``.xml`` and ``.bin`` files),
* the `Python API functions <#convert-a-model-with-python-convert-model>`__ (``openvino.convert_model`` and ``openvino.save_model``)
* the `ovc <#convert-a-model-in-cli-ovc>`__ command line tool.
Once saved as :doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>` (a set of ``.xml`` and ``.bin`` files),
the model may be deployed with maximum performance. Because it is already optimized
for OpenVINO inference, it can be read, compiled, and inferred with no additional delay.
for `OpenVINO inference <openvino_docs_OV_UG_Integrate_OV_with_your_application>`__, it can be read, compiled, and inferred with no additional delay.
.. note::
@ -51,13 +48,37 @@ and `Torchvision models <https://pytorch.org/hub/>`__. Now you have two options:
with ``openvino.tools.mo.convert_model`` or the ``mo`` CLI tool.
For more details, see the :doc:`Model Conversion API Transition Guide <openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition>`.
For PyTorch models, `Python API <#convert-a-model-with-python-convert-model>`__ is the only conversion option.
TensorFlow may present additional considerations :doc:`TensorFlow Frontend Capabilities and Limitations <openvino_docs_MO_DG_TensorFlow_Frontend>`.
Model States
##############################################
There are three states a model in OpenVINO can be: saved on disk, loaded but not compiled (``ov.Model``) or loaded and compiled (``ov.CompiledModel``).
| **Saved on disk**
| A model in this state consists of one or more files that fully represent the neural network. A model can be stored in different ways. For example:
| OpenVINO IR: pair of .xml and .bin files
| ONNX: .onnx file
| TensorFlow: directory with a .pb file and two subfolders or just a .pb file
| TensorFlow Lite: .tflite file
| PaddlePaddle: .pdmodel file
| **Loaded but not compiled**
| A model object (``ov.Model``) is created in memory either by parsing a file or converting an existing framework object. Inference cannot be done with this object yet as it is not attached to any specific device, but it allows customization such as reshaping its input, applying quantization or even adding preprocessing steps before compiling the model.
| **Loaded and compiled**
| This state is achieved when one or more devices are specified for a model object to run on (``ov.CompiledModel``), allowing device optimizations to be made and enabling inference.
For more information on each function, see the :doc:`OpenVINO workflow <openvino_workflow>` page.
Convert a Model with Python: ``convert_model``
##############################################
The Model conversion API in Python uses the ``openvino.convert_model`` function,
turning a given model into the `openvino.Model <api/ie_python_api/_autosummary/openvino.runtime.Model.html>`__
object and loading it to memory. Now it can be: saved to a drive with `openvino.save_model``
object and loading it to memory. Now it can be: saved to a drive with ``openvino.save_model``
or further :doc:`optimized with NNCF <openvino_docs_model_optimization_guide>`
prior to saving.
@ -233,21 +254,18 @@ Convert a Model in CLI: ``ovc``
``ovc`` is a command-line model converter, combining the ``openvino.convert_model``
and ``openvino.save_model`` functionalities, providing the exact same results, if the same set of
parameters is used for saving into OpenVINO IR. It converts files from one of the
:doc:`supported model formats <Supported_Model_Formats>` to :doc:`OpenVINO IR <openvino_ir>`, which can then be read, compiled,
parameters is used for saving into OpenVINO IR. It converts files from one of the to :doc:`OpenVINO IR <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`, which can then be read, compiled,
and run by the final inference application.
.. note::
PyTorch models cannot be converted with ``ovc``, use ``openvino.convert_model`` instead.
Additional Resources
####################
The following articles describe in detail how to obtain and prepare your model depending on the source model type:
* :doc:`Convert different model formats to the ov.Model format <Supported_Model_Formats>`.
* :doc:`Convert different model formats to the ov.Model format <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR>`.
* :doc:`Review all available conversion parameters <openvino_docs_OV_Converter_UG_Conversion_Options>`.
To achieve the best model inference performance and more compact OpenVINO IR representation follow:

142
docs/articles_en/openvino_workflow/model_preparation/supported_model_formats.rst → docs/articles_en/openvino_workflow/model_preparation/Convert_Model_IR.rst
View File

@ -1,54 +1,28 @@
.. {#Supported_Model_Formats}
.. {#openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_IR}
Supported Model Formats
=======================
Convert to OpenVINO IR
=============================================
.. meta::
:description: Learn about supported model formats and the methods used to convert, read, and compile them in OpenVINO™.
| **OpenVINO IR (Intermediate Representation)**
| The proprietary format of OpenVINO™, benefiting from the full extent of its features.
It is obtained by :doc:`converting a model <openvino_docs_model_processing_introduction>`
from one of the remaining supported formats using the Python model conversion API or the
OpenVINO Converter.
| Consider storing your model in this format to minimize first-inference latency,
perform model optimizations, and save space on your drive, in some cases.
| **PyTorch, TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle**
| These supported model formats can be read, compiled, and converted to OpenVINO IR,
either automatically or explicitly.
In the Python API, these options are provided as three separate methods:
``read_model()``, ``compile_model()``, and ``convert_model()``.
The ``convert_model()`` method enables you to perform additional adjustments
to the model, such as setting shapes, changing model input types or layouts,
cutting parts of the model, freezing inputs, etc. For a detailed description
of the conversion process, see the
:doc:`model conversion guide <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
Note that for PyTorch models, Python API
is the only conversion option.
TensorFlow may present additional considerations
:doc:`TensorFlow Frontend Capabilities and Limitations <openvino_docs_MO_DG_TensorFlow_Frontend>`.
:description: Convert models from the original framework to OpenVINO representation.
.. toctree::
:maxdepth: 1
:hidden:
Convert from PyTorch <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch>
Convert from TensorFlow <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow>
Convert from ONNX <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_ONNX>
Convert from TensorFlow_Lite <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite>
Convert from PaddlePaddle <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_Paddle>
:doc:`IR (Intermediate Representation) <openvino_ir>` is OpenVINO own format consisting of ``.xml`` and ``.bin`` files.
Convert the model into OpenVINO IR for `better performance <#ir-conversion-benefits>`__.
Convert Models
##############################################
Here are code examples of how to use these methods with different model formats:
@ -557,49 +531,57 @@ Here are code examples of how to use these methods with different model formats:
:doc:`article <openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_Paddle>`.
* :doc:`How to convert PyTorch <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_PyTorch>`
* :doc:`How to convert ONNX <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_ONNX>`
* :doc:`How to convert TensorFlow <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow>`
* :doc:`How to convert TensorFlow Lite <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_TensorFlow_Lite>`
* :doc:`How to convert PaddlePaddle <openvino_docs_OV_Converter_UG_prepare_model_convert_model_Convert_Model_From_Paddle>`
To choose the best workflow for your application, read the :doc:`Model Preparation section <openvino_docs_model_processing_introduction>`
To choose the best workflow for your application, read the :doc:`Model Preparation section <openvino_docs_model_processing_introduction>`.
Refer to the list of all supported conversion options in :doc:`Conversion Parameters <openvino_docs_OV_Converter_UG_Conversion_Options>`
Refer to the list of all supported conversion options in :doc:`Conversion Parameters <openvino_docs_OV_Converter_UG_Conversion_Options>`.
IR Conversion Benefits
################################################
| **Saving to IR to improve first inference latency**
| When first inference latency matters, rather than convert the framework model each time it is loaded, which may take some time depending on its size, it is better to do it once. Save the model as an OpenVINO IR with ``save_model`` and then load it with ``read_model`` as needed. This should improve the time it takes the model to make the first inference as it avoids the conversion step.
| **Saving to IR in FP16 to save space**
| Save storage space, even more so if FP16 is used as it may cut the size by about 50%, especially useful for large models, like Llama2-7B.
| **Saving to IR to avoid large dependencies in inference code**
| Frameworks such as TensorFlow and PyTorch tend to be large dependencies (multiple gigabytes), and not all inference environments have enough space to hold them.
| Converting models to OpenVINO IR allows them to be used in an environment where OpenVINO is the only dependency, so much less disk space is needed.
| Loading and compiling with OpenVINO directly usually takes less runtime memory than loading the model in the source framework and then converting and compiling it.
An example showing how to take advantage of OpenVINO IR, saving a model in OpenVINO IR once, using it many times, is shown below:
.. code-block:: py
# Run once
import openvino as ov
import tensorflow as tf
# 1. Convert model created with TF code
model = tf.keras.applications.resnet50.ResNet50(weights="imagenet")
ov_model = ov.convert_model(model)
# 2. Save model as OpenVINO IR
ov.save_model(ov_model, 'model.xml', compress_to_fp16=True) # enabled by default
# Repeat as needed
import openvino as ov
# 3. Load model from file
core = ov.Core()
ov_model = core.read_model("model.xml")
# 4. Compile model from memory
compiled_model = core.compile_model(ov_model)
Additional Resources
####################

2
docs/articles_en/openvino_workflow/model_preparation/conversion_parameters.rst
View File

@ -7,7 +7,7 @@ Conversion Parameters
.. meta::
:description: Model Conversion API provides several parameters to adjust model conversion.
This document describes all available parameters for ``openvino.convert_model``, ``ovc``, and ``openvino.save_model`` without focusing on a particular framework model format. Use this information for your reference as a common description of the conversion API capabilities in general. Part of the options can be not relevant to some specific frameworks. Use :doc:`Supported Model Formats <Supported_Model_Formats>` page for more dedicated framework-dependent tutorials.
This document describes all available parameters for ``openvino.convert_model``, ``ovc``, and ``openvino.save_model`` without focusing on a particular framework model format. Use this information for your reference as a common description of the conversion API capabilities in general. Part of the options can be not relevant to some specific frameworks. Use :doc:`Supported Model Formats <openvino_docs_model_processing_introduction>` page for more dedicated framework-dependent tutorials.
In most cases when it is required to convert a model the following simple syntax can be used:

12
docs/articles_en/openvino_workflow/model_preparation/pytorch_vision.rst Normal file
View File

@ -0,0 +1,12 @@
.. {#pytorch_vision}
PyVision
=======================
.. meta::
:description: Learn about supported model formats and the methods used to convert, read, and compile them in OpenVINO™.
Images input to AI models often need to be preprocessed in order to have proper dimensions or data type.
Instead of doing it with another library in an additional pipeline step, you can use torchvision.transforms OpenVINO feature.
It automatically translates a torchvision preprocessing pipeline to OpenVINO operators and then embeds them into your OpenVINO model, reducing overall program complexity and allowing additional performance optimizations to take place.

229
docs/articles_en/openvino_workflow/model_preparation_tensorflow_pytorch_guide.rst Normal file
View File

@ -0,0 +1,229 @@
.. {#openvino_docs_model_processing_introduction_draft}
Import TensorFlow and PyTorch Models
==============================================
In OpenVINO you can load a model in different formats.
The examples below show how TensorFlow and PyTorch models. The models are loaded, converted to OpenVINO format, and compiled for inferencing in just several lines of code.
To learn more about how models can be imported in OpenVINO, refer to the :doc:`Model Preparation <openvino_docs_model_processing_introduction>` page.
.. tab-set::
.. tab-item:: Import TensorFlow model
.. code-block:: py
:force:
import openvino as ov
# 1. Compile model from file
core = ov.Core()
compiled_model = core.compile_model("model.pb")
.. tab-item:: Import PyTorch model
.. code-block:: py
import openvino as ov
import torch
# 1. Convert model loaded from PyTorch file
model = torch.load("model.pt")
model.eval()
ov_model = ov.convert_model(model)
# 2. Compile model from memory
core = ov.Core()
compiled_model = core.compile_model(ov_model)
While the above examples provide a simple and straightforward option to import models into OpenVINO, there are other options to provide more customization and flexibility.
TensorFlow Import Options
##############################################
OpenVINO direct support of TensorFlow allows developers to use their models in an OpenVINO inference pipeline without changes. However, as multiple ways of doing this exist, it may not be clear which is the best approach for a given situation. The following diagram aims to simplify this decision given a certain context, although some additional considerations should be taken into account depending on the use case.
.. image:: _static/images/import_tensorflow.svg
Method 1. Convert using ov.convert_model function (Python only)
---------------------------------------------------------------------
As seen above, if your starting point is a Python object in memory, for example a ``tf.keras.Model`` or ``tf.Module``, a direct way to get the model in OpenVINO is to use ``ov.convert_model``. This method produces an ``ov.Model`` (one of the three states) that can later be reshaped, saved to OpenVINO IR or compiled to do inference. In code it may look as follows:
.. code-block:: py
import openvino as ov
import tensorflow as tf
# 1a. Convert model created with TF code
model = tf.keras.applications.resnet50.ResNet50(weights="imagenet")
ov_model = ov.convert_model(model)
# 1b. Convert model from file
ov_model = ov.convert_model("model.pb")
# 2. Compile model from memory
core = ov.Core()
compiled_model = core.compile_model(ov_model)
Method 2. Convert from file using ov.compile_model function
---------------------------------------------------------------------
In case you are starting with a file, you will need to see if the needs to be customized, such as applying quantization or reshaping its inputs.
If the model does not need to be customized, ``ov.Core.compile_model`` should be used, which reads, converts (if needed) and compiles the model, leaving it ready for inference all in one go. The code should look like this:
.. code-block:: py
import openvino as ov
# 1. Compile model from file
core = ov.Core()
compiled_model = core.compile_model("model.pb")
Method 3. Convert from file using ov.read_model function
---------------------------------------------------------------------
If the model does need to be customized, ``ov.read_model`` can be used as it just returns an ``ov.Model`` ready to be quantized or have its inputs reshaped. (Note: This method also works with the OpenVINO C++ API, so it is useful for developers working in a C++ environment.)
.. code-block:: py
import openvino as ov
# 1. Convert model from file
core = ov.Core()
ov_model = ov.read_model("model.pb")
# 2. Compile model from memory
compiled_model = core.compile_model(ov_model)
Method 4. Convert from file using OpenVINO Model Converter (ovc CLI)
---------------------------------------------------------------------
However, if the input reshaping is known in advance and/or the model has multiple outputs but only some of them are required, OpenVINO provides two equivalent ways of doing these while converting the model. One of them is the CLI command ``ovc`` while the other is the previously mentioned ``ov.convert_model`` (Method 1).
The ``ovc`` tool is similar to ``ov.convert_model``, except it works using the command line rather than a Python environment. It will convert the model to OpenVINO IR format, apply any configurations you specify, and save the converted model to disk. It is useful if you are not working with your model in Python (e.g., if you are developing in a C++ environment) or if you prefer using the command line rather than a Python script.
The code below shows how to convert a model with ovc and then load it for inference:
.. code-block:: py
# 1. Convert model from file
ovc model.pb
.. code-block:: py
import openvino as ov
# 2. Load model from file
core = ov.Core()
ov_model = core.read_model("model.xml")
# 3. Compile model from memory
compiled_model = core.compile_model(ov_model)
PyTorch Import Options
##############################################
OpenVINO direct support of PyTorch allows developers to use their models in an OpenVINO inference pipeline without changes. OpenVINO provides multiple ways of using PyTorch. The following diagram aims to simplify this decision given a certain context, although some additional considerations should be taken into account depending on the use case.
.. image:: _static/images/import_pytorch.svg
PyTorch models can be imported into OpenVINO directly from a Python object. Saved PyTorch files can be used as well. To use a saved PyTorch file, it needs to be loaded in PyTorch first to convert it to a Python object.
Once the model is loaded as a PyTorch Python object, you can decide whether to start using the OpenVINO framework and its features directly or to remain within the PyTorch framework while leveraging optimizations.
Method 1. Convert using ov.convert_model function
---------------------------------------------------------------------
If OpenVINO is preferred, ov.convert_model is the method to use. It produces an ``ov.Model`` that can later be reshaped, saved to OpenVINO IR or compiled to do inference. In code it may look as follows:
.. code-block:: py
import openvino as ov
import torch
from torchvision.models import resnet50
# 1a. Convert model created with PyTorch code
model = resnet50(weights="DEFAULT")
model.eval()
ov_model = ov.convert_model(model, example_input=torch.rand(1, 3, 224, 224))
# 1b. Convert model loaded from PyTorch file
model = torch.load("model.pt")
model.eval()
ov_model = ov.convert_model(model)
# 2. Compile model from memory
core = ov.Core()
compiled_model = core.compile_model(ov_model)
Note that the need to set ``example_input`` depends on the model used. However, it is recommended to always set it if available as it usually leads to a better quality model. For more details, check out the docs.
Method 2. Use OpenVINO backend in PyTorch
---------------------------------------------------------------------
In case PyTorch syntax is preferred, since PyTorch 2.0 and OpenVINO 2023.1, a PyTorch model can be optimized with OpenVINO by specifying it as a backend in ``torch.compile``.
.. code-block:: py
import openvino.torch
import torch
from torchvision.models import resnet50
# 1a. Compile model created with PyTorch code
model = resnet50(weights="DEFAULT")
model.eval()
compiled_model = torch.compile(model, backend="openvino")
# 1b. Compile model loaded from PyTorch file
model = torch.load("model.pt")
model.eval()
compiled_model = torch.compile(model, backend="openvino")
Method 3. Export model to ONNX and use one of OpenVINO methods
---------------------------------------------------------------------
If none of these two methods convert the model successfully, there is a third method that once was the main way of using PyTorch in OpenVINO, but now is mainly considered a backup plan.
This method consists of exporting a PyTorch model to ONNX and then loading it with the different methods available in OpenVINO. See ONNX, PaddlePaddle and TensorFlow Lite Import Options for more details.
.. code-block:: py
import torch
import openvino as ov
from torchvision.models import resnet50
# 1. Export PyTorch model to ONNX
model = resnet50(weights="DEFAULT")
model.eval()
dummy_input = torch.randn(1,3,224,224)
torch.onnx.export(model, dummy_input, "model.onnx")
# 2. Use an OpenVINO method to read and compile it, for example compile_model
core = ov.Core()
compiled_model = core.compile_model("model.onnx")
Supported Model Formats
---------------------------------------------------------------------
As PyTorch does not have a save format that contains everything needed to reproduce the model without using torch, OpenVINO only supports loading Python objects directly. The support is as follows:
* Python objects
* torch.nn.Module
* torch.jit.ScriptModule
* torch.jit.ScriptFunction
Jupyter Notebook Tutorials
################################################
OpenVINO also provides example notebooks for both frameworks showing how to load a model and make inference:
* `Convert TensorFlow Models to OpenVINO <notebooks/101-tensorflow-classification-to-openvino-with-output.html>`__
* `Convert PyTorch Models to OpenVINO <notebooks/102-pytorch-onnx-to-openvino-with-output.html>`__

4
docs/articles_en/openvino_workflow/running_inference_with_openvino.rst
View File

@ -22,7 +22,7 @@ Running Inference with OpenVINO™
OpenVINO Runtime is a set of C++ libraries with C and Python bindings providing a common API
to deploy inference on the platform of your choice. You can run any of the
:doc:`supported model formats <Supported_Model_Formats>` directly or convert the model
:doc:`supported model formats <openvino_docs_model_processing_introduction>` directly or convert the model
and save it to the :doc:`OpenVINO IR <openvino_ir>` format, for maximum performance.
Why is OpenVINO IR inference faster? Even if you run a supported model directly, it is
@ -37,7 +37,7 @@ OpenVINO IR provides by far the best first-inference latency scores.
.. note::
For more detailed information on how to convert, read, and compile supported model formats
see the :doc:`Supported Formats article <Supported_Model_Formats>`.
see the :doc:`Model Preparation article <openvino_docs_model_processing_introduction>`.
Note that TensorFlow models can be run using the
:doc:`torch.compile feature <pytorch_2_0_torch_compile>`, as well as the standard ways of

18
docs/home.rst
View File

@ -61,40 +61,40 @@ OpenVINO 2023.2
See latest benchmark numbers for OpenVINO and OpenVINO Model Server
.. grid-item-card:: Flexible Workflow
:link: Supported_Model_Formats
.. grid-item-card:: Work with Multiple Model Formats
:link: openvino_docs_model_processing_introduction
:link-alt: Supported Model Formats
:link-type: doc
Load models directly (for TensorFlow, ONNX, PaddlePaddle) or convert to the OpenVINO format.
OpenVINO supports different model formats: PyTorch, TensorFlow, TensorFlow Lite, ONNX, and PaddlePaddle.
.. grid-item-card:: Deploy at Scale With OpenVINO Model Server
.. grid-item-card:: Deploy at Scale with OpenVINO Model Server
:link: ovms_what_is_openvino_model_server
:link-alt: model server
:link-type: doc
Cloud-ready deployments for microservice applications
.. grid-item-card:: Model Optimization
.. grid-item-card:: Optimize Models
:link: openvino_docs_model_optimization_guide
:link-alt: model optimization
:link-type: doc
Reach for performance with post-training and training-time compression with NNCF
Boost performance using quantization and compression with NNCF
.. grid-item-card:: PyTorch 2.0 - torch.compile() backend
.. grid-item-card:: Use OpenVINO with PyTorch Apps with torch.compile()
:link: pytorch_2_0_torch_compile
:link-alt: torch.compile
:link-type: doc
Optimize generation of the graph model with PyTorch 2.0 torch.compile() backend
.. grid-item-card:: Generative AI optimization and deployment
.. grid-item-card:: Optimize and Deploy Generative AI
:link: gen_ai_guide
:link-alt: gen ai
:link-type: doc
Generative AI optimization and deployment
Enhance the efficiency of Generative AI
Feature Overview

3
docs/sphinx_setup/_static/images/img/import_pytorch.svg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7419b60d37a9bc058626c52fcbfec20c3a5d22c6d0875fb84ef0df7ec2a68671
size 142191

3
docs/sphinx_setup/_static/images/img/import_tensorflow.svg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5666c2ee7503bc2844a99f73c1b64afacd2c42dadef441ce115cc18b00922c7
size 224644

3
docs/sphinx_setup/_static/images/import_pytorch.svg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7419b60d37a9bc058626c52fcbfec20c3a5d22c6d0875fb84ef0df7ec2a68671
size 142191

3
docs/sphinx_setup/_static/images/import_tensorflow.svg Normal file
View File

@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d5666c2ee7503bc2844a99f73c1b64afacd2c42dadef441ce115cc18b00922c7
size 224644