IntenseWebs/openvino
3
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS shift to rst (#16686)

Browse Source
This commit is contained in:
Sebastian Golebiewski authored and GitHub committed 2023-04-03 11:41:28 +02:00
1 parent 016d36f032
commit e7c1cdf982
11 files changed
+719 -518
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/MO_DG/prepare_model/convert_model/Convert_Model_From_PyTorch.md
+4 -4
View File
@@ -4,10 +4,10 @@
The PyTorch framework is supported through export to the ONNX format. In order to optimize and deploy a model that was trained with it:
1. `Export a PyTorch model to ONNX <#Exporting-a-PyTorch-Model-to-ONNX-Format>`__.
1. `Export a PyTorch model to ONNX <#exporting-a-pytorch-model-to-onnx-format>`__.
2. :doc:`Convert the ONNX model <openvino_docs_MO_DG_prepare_model_convert_model_Convert_Model_From_ONNX>` to produce an optimized :doc:`Intermediate Representation <openvino_docs_MO_DG_IR_and_opsets>` of the model based on the trained network topology, weights, and biases values.
Exporting a PyTorch Model to ONNX Format
Exporting a PyTorch Model to ONNX Format
########################################
PyTorch models are defined in Python. To export them, use the ``torch.onnx.export()`` method. The code to
@@ -17,7 +17,7 @@ For more information, refer to the `Exporting PyTorch models to ONNX format <htt
To export a PyTorch model, you need to obtain the model as an instance of ``torch.nn.Module`` class and call the ``export`` function.
.. code-block:: python
.. code-block:: py
import torch
@@ -32,7 +32,7 @@ To export a PyTorch model, you need to obtain the model as an instance of ``torc
Known Issues
############
####################
As of version 1.8.1, not all PyTorch operations can be exported to ONNX opset 9 which is used by default.
It is recommended to export models to opset 11 or higher when export to default opset 9 is not working. In that case, use ``opset_version`` option of the ``torch.onnx.export``. For more information about ONNX opset, refer to the `Operator Schemas <https://github.com/onnx/onnx/blob/master/docs/Operators.md>`__ page.
docs/MO_DG/prepare_model/convert_model/kaldi_specific/Aspire_Tdnn_Model.md
+111 -76
View File
@@ -1,110 +1,145 @@
# Converting a Kaldi ASpIRE Chain Time Delay Neural Network (TDNN) Model {#openvino_docs_MO_DG_prepare_model_convert_model_kaldi_specific_Aspire_Tdnn_Model}
At the beginning, you should [download a pre-trained model](https://kaldi-asr.org/models/1/0001_aspire_chain_model.tar.gz)
@sphinxdirective
At the beginning, you should `download a pre-trained model <https://kaldi-asr.org/models/1/0001_aspire_chain_model.tar.gz>`__
for the ASpIRE Chain Time Delay Neural Network (TDNN) from the Kaldi project official website.
## Converting an ASpIRE Chain TDNN Model to IR
Converting an ASpIRE Chain TDNN Model to IR
###########################################
Generate the Intermediate Representation of the model by running Model Optimizer with the following parameters:
```sh
mo --input_model exp/chain/tdnn_7b/final.mdl --output output
```
The IR will have two inputs: `input` for data, and `ivector` for ivectors.
.. code-block:: sh
## Example: Running ASpIRE Chain TDNN Model with the Speech Recognition Sample
mo --input_model exp/chain/tdnn_7b/final.mdl --output output
> **NOTE**: Before you continue with this part of the article, get familiar with the [Speech Recognition sample](../../../../../samples/cpp/speech_sample/README.md).
In this example, the input data contains one utterance from one speaker.
The IR will have two inputs: ``input`` for data, and ``ivector`` for ivectors.
Example: Running ASpIRE Chain TDNN Model with the Speech Recognition Sample
###########################################################################
.. note::
Before you continue with this part of the article, get familiar with the
:doc:`Speech Recognition sample <openvino_inference_engine_samples_speech_sample_README>`.
In this example, the input data contains one utterance from one speaker.
To run the ASpIRE Chain TDNN Model with Speech Recognition sample, You need to prepare environment. Do it by following the steps below :
1. Download a [Kaldi repository](https://github.com/kaldi-asr/kaldi).
2. Build it by following instructions in `README.md` from the repository.
3. Download the [model archive](https://kaldi-asr.org/models/1/0001_aspire_chain_model.tar.gz) from Kaldi website.
4. Extract the downloaded model archive to the `egs/aspire/s5` folder of the Kaldi repository.
1. Download a `Kaldi repository <https://github.com/kaldi-asr/kaldi>`__.
2. Build it by following instructions in ``README.md`` from the repository.
3. Download the `model archive <https://kaldi-asr.org/models/1/0001_aspire_chain_model.tar.gz>`__ from Kaldi website.
4. Extract the downloaded model archive to the ``egs/aspire/s5`` folder of the Kaldi repository.
Once everything has been prepared, you can start a proper run:
1. Prepare the model for decoding. Refer to the `README.txt` file from the downloaded model archive for instructions.
2. Convert data and ivectors to `.ark` format. Refer to the corresponding sections below for instructions.
1. Prepare the model for decoding. Refer to the ``README.txt`` file from the downloaded model archive for instructions.
2. Convert data and ivectors to ``.ark`` format. Refer to the corresponding sections below for instructions.
### Preparing Data
Preparing Data
++++++++++++++++++++
If you have a `.wav` data file, convert it to the `.ark` format using the following command:
```sh
<path_to_kaldi_repo>/src/featbin/compute-mfcc-feats --config=<path_to_kaldi_repo>/egs/aspire/s5/conf/mfcc_hires.conf scp:./wav.scp ark,scp:feats.ark,feats.scp
```
Add the `feats.ark` absolute path to `feats.scp` to avoid errors in later commands.
If you have a ``.wav`` data file, convert it to the ``.ark`` format using the following command:
### Preparing Ivectors
.. code-block:: sh
<path_to_kaldi_repo>/src/featbin/compute-mfcc-feats --config=<path_to_kaldi_repo>/egs/aspire/s5/conf/mfcc_hires.conf scp:./wav.scp ark,scp:feats.ark,feats.scp
Add the ``feats.ark`` absolute path to ``feats.scp`` to avoid errors in later commands.
Preparing Ivectors
++++++++++++++++++++
Prepare ivectors for the Speech Recognition sample:
1. Copy the `feats.scp` file to the `egs/aspire/s5/` directory of the built Kaldi repository and navigate there:
```sh
cp feats.scp <path_to_kaldi_repo>/egs/aspire/s5/
cd <path_to_kaldi_repo>/egs/aspire/s5/
```
1. Copy the ``feats.scp`` file to the ``egs/aspire/s5/`` directory of the built Kaldi repository and navigate there:
.. code-block:: sh
cp feats.scp <path_to_kaldi_repo>/egs/aspire/s5/
cd <path_to_kaldi_repo>/egs/aspire/s5/
2. Extract ivectors from the data:
```sh
./steps/online/nnet2/extract_ivectors_online.sh --nj 1 --ivector_period <max_frame_count_in_utterance> <data folder> exp/tdnn_7b_chain_online/ivector_extractor <ivector folder>
```
You can simplify the preparation of ivectors for the Speech Recognition sample. To do it, specify the maximum number of frames in utterances as a parameter for `--ivector_period`
to get only one ivector per utterance.
To get the maximum number of frames in utterances, use the following command line:
```sh
../../../src/featbin/feat-to-len scp:feats.scp ark,t: | cut -d' ' -f 2 - | sort -rn | head -1
```
As a result, you will find the `ivector_online.1.ark` file in `<ivector folder>`.
.. code-block:: sh
3. Go to the `<ivector folder>`:
```sh
cd <ivector folder>
```
./steps/online/nnet2/extract_ivectors_online.sh --nj 1 --ivector_period <max_frame_count_in_utterance> <data folder> exp/tdnn_7b_chain_online/ivector_extractor <ivector folder>
4. Convert the `ivector_online.1.ark` file to text format, using the `copy-feats` tool. Run the following command:
```sh
<path_to_kaldi_repo>/src/featbin/copy-feats --binary=False ark:ivector_online.1.ark ark,t:ivector_online.1.ark.txt
```
5. For the Speech Recognition sample, the `.ark` file must contain an ivector
for each frame. Copy the ivector `frame_count` times by running the below script in the Python command prompt:
```python
import subprocess
You can simplify the preparation of ivectors for the Speech Recognition sample. To do it, specify the maximum number of frames in utterances as a parameter for ``--ivector_period`` to get only one ivector per utterance.
subprocess.run(["<path_to_kaldi_repo>/src/featbin/feat-to-len", "scp:<path_to_kaldi_repo>/egs/aspire/s5/feats.scp", "ark,t:feats_length.txt"])
To get the maximum number of frames in utterances, use the following command line:
f = open("ivector_online.1.ark.txt", "r")
g = open("ivector_online_ie.ark.txt", "w")
length_file = open("feats_length.txt", "r")
for line in f:
if "[" not in line:
for i in range(frame_count):
line = line.replace("]", " ")
g.write(line)
else:
g.write(line)
frame_count = int(length_file.read().split(" ")[1])
g.write("]")
f.close()
g.close()
length_file.close()
```
.. code-block:: sh
6. Create an `.ark` file from `.txt`:
```sh
<path_to_kaldi_repo>/src/featbin/copy-feats --binary=True ark,t:ivector_online_ie.ark.txt ark:ivector_online_ie.ark
```
../../../src/featbin/feat-to-len scp:feats.scp ark,t: | cut -d' ' -f 2 - | sort -rn | head -1
### Running the Speech Recognition Sample
Run the Speech Recognition sample with the created ivector `.ark` file:
```sh
speech_sample -i feats.ark,ivector_online_ie.ark -m final.xml -d CPU -o prediction.ark -cw_l 17 -cw_r 12
```
As a result, you will find the ``ivector_online.1.ark`` file in ``<ivector folder>``.
3. Go to the ``<ivector folder>``:
.. code-block:: sh
cd <ivector folder>
4. Convert the ``ivector_online.1.ark`` file to text format, using the ``copy-feats`` tool. Run the following command:
.. code-block:: sh
<path_to_kaldi_repo>/src/featbin/copy-feats --binary=False ark:ivector_online.1.ark ark,t:ivector_online.1.ark.txt
5. For the Speech Recognition sample, the ``.ark`` file must contain an ivector
for each frame. Copy the ivector ``frame_count`` times by running the below script in the Python command prompt:
.. code-block:: python
import subprocess
subprocess.run(["<path_to_kaldi_repo>/src/featbin/feat-to-len", "scp:<path_to_kaldi_repo>/egs/aspire/s5/feats.scp", "ark,t:feats_length.txt"])
f = open("ivector_online.1.ark.txt", "r")
g = open("ivector_online_ie.ark.txt", "w")
length_file = open("feats_length.txt", "r")
for line in f:
if "[" not in line:
for i in range(frame_count):
line = line.replace("]", " ")
g.write(line)
else:
g.write(line)
frame_count = int(length_file.read().split(" ")[1])
g.write("]")
f.close()
g.close()
length_file.close()
6. Create an ``.ark`` file from ``.txt``:
.. code-block:: sh
<path_to_kaldi_repo>/src/featbin/copy-feats --binary=True ark,t:ivector_online_ie.ark.txt ark:ivector_online_ie.ark
Running the Speech Recognition Sample
+++++++++++++++++++++++++++++++++++++
Run the Speech Recognition sample with the created ivector ``.ark`` file:
.. code-block:: sh
speech_sample -i feats.ark,ivector_online_ie.ark -m final.xml -d CPU -o prediction.ark -cw_l 17 -cw_r 12
Results can be decoded as described in "Use of Sample in Kaldi Speech Recognition Pipeline"
in the [Speech Recognition Sample description](../../../../../samples/cpp/speech_sample/README.md) article.
in the :doc:`Speech Recognition Sample description <openvino_inference_engine_samples_speech_sample_README>` article.
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/mxnet_specific/Convert_GluonCV_Models.md
+36 -21
View File
@@ -1,26 +1,41 @@
# Converting MXNet GluonCV Models {#openvino_docs_MO_DG_prepare_model_convert_model_mxnet_specific_Convert_GluonCV_Models}
This article provides the instructions and examples on how to use Model Optimizer to convert [GluonCV SSD and YOLO-v3 models](https://gluon-cv.mxnet.io/model_zoo/detection.html) to IR.
@sphinxdirective
This article provides the instructions and examples on how to use Model Optimizer to convert `GluonCV SSD and YOLO-v3 models <https://gluon-cv.mxnet.io/model_zoo/detection.html>`__ to IR.
1. Choose the topology available from the `GluonCV Model Zoo <https://gluon-cv.mxnet.io/model_zoo/detection.html>`__ and export to the MXNet format using the GluonCV API. For example, for the ``ssd_512_mobilenet1.0`` topology:
.. code-block:: python
from gluoncv import model_zoo, data, utils
from gluoncv.utils import export_block
net = model_zoo.get_model('ssd_512_mobilenet1.0_voc', pretrained=True)
export_block('ssd_512_mobilenet1.0_voc', net, preprocess=True, layout='HWC')
As a result, you will get an MXNet model representation in ``ssd_512_mobilenet1.0.params`` and ``ssd_512_mobilenet1.0.json`` files generated in the current directory.
2. Run the Model Optimizer tool, specifying the ``--enable_ssd_gluoncv`` option. Make sure the ``--input_shape`` parameter is set to the input shape layout of your model (NHWC or NCHW). The examples below illustrate running the Model Optimizer for the SSD and YOLO-v3 models trained with the NHWC layout and located in the ``<model_directory>``:
1. Choose the topology available from the [GluonCV Model Zoo](https://gluon-cv.mxnet.io/model_zoo/detection.html) and export to the MXNet format using the GluonCV API. For example, for the `ssd_512_mobilenet1.0` topology:
```python
from gluoncv import model_zoo, data, utils
from gluoncv.utils import export_block
net = model_zoo.get_model('ssd_512_mobilenet1.0_voc', pretrained=True)
export_block('ssd_512_mobilenet1.0_voc', net, preprocess=True, layout='HWC')
```
As a result, you will get an MXNet model representation in `ssd_512_mobilenet1.0.params` and `ssd_512_mobilenet1.0.json` files generated in the current directory.
2. Run the Model Optimizer tool, specifying the `--enable_ssd_gluoncv` option. Make sure the `--input_shape` parameter is set to the input shape layout of your model (NHWC or NCHW). The examples below illustrate running the Model Optimizer for the SSD and YOLO-v3 models trained with the NHWC layout and located in the `<model_directory>`:
* **For GluonCV SSD topologies:**
```sh
mo --input_model <model_directory>/ssd_512_mobilenet1.0.params --enable_ssd_gluoncv --input_shape [1,512,512,3] --input data --output_dir <OUTPUT_MODEL_DIR>
```
.. code-block:: sh
mo --input_model <model_directory>/ssd_512_mobilenet1.0.params --enable_ssd_gluoncv --input_shape [1,512,512,3] --input data --output_dir <OUTPUT_MODEL_DIR>
* **For YOLO-v3 topology:**
* To convert the model:
```sh
mo --input_model <model_directory>/yolo3_mobilenet1.0_voc-0000.params --input_shape [1,255,255,3] --output_dir <OUTPUT_MODEL_DIR>
```
* To convert the model with replacing the subgraph with RegionYolo layers:
```sh
mo --input_model <model_directory>/models/yolo3_mobilenet1.0_voc-0000.params --input_shape [1,255,255,3] --transformations_config "front/mxnet/yolo_v3_mobilenet1_voc.json" --output_dir <OUTPUT_MODEL_DIR>
```
* To convert the model:
.. code-block:: sh
mo --input_model <model_directory>/yolo3_mobilenet1.0_voc-0000.params --input_shape [1,255,255,3] --output_dir <OUTPUT_MODEL_DIR>
* To convert the model with replacing the subgraph with RegionYolo layers:
.. code-block:: sh
mo --input_model <model_directory>/models/yolo3_mobilenet1.0_voc-0000.params --input_shape [1,255,255,3] --transformations_config "front/mxnet/yolo_v3_mobilenet1_voc. json" --output_dir <OUTPUT_MODEL_DIR>
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/mxnet_specific/Convert_Style_Transfer_From_MXNet.md
+145 -91
View File
@@ -1,117 +1,171 @@
# Converting an MXNet Style Transfer Model {#openvino_docs_MO_DG_prepare_model_convert_model_mxnet_specific_Convert_Style_Transfer_From_MXNet}
@sphinxdirective
This article provides instructions on how to generate a model for style transfer, using the public MXNet neural style transfer sample.
**Step 1**: Download or clone the repository [Zhaw's Neural Style Transfer repository](https://github.com/zhaw/neural_style) with an MXNet neural style transfer sample.
**Step 1**: Download or clone the repository `Zhaw's Neural Style Transfer repository <https://github.com/zhaw/neural_style>`__ with an MXNet neural style transfer sample.
**Step 2**: Prepare the environment required to work with the cloned repository:
> **NOTE**: Python-tk installation is needed only for Linux. Python for Windows includes it by default.
.. note::
Python-tk installation is needed only for Linux. Python for Windows includes it by default.
1. Install packages dependency.
.. code-block:: sh
sudo apt-get install python-tk
1. Install packages dependency.<br>
```sh
sudo apt-get install python-tk
```
2. Install Python requirements:
```sh
pip3 install --user mxnet
pip3 install --user matplotlib
pip3 install --user scikit-image
```
**Step 3**: Download the pretrained [VGG19 model](https://github.com/dmlc/web-data/raw/master/mxnet/neural-style/model/vgg19.params) and save it to the root directory of the cloned repository. The sample expects the model `vgg19.params` file to be in that directory.<br>
.. code-block:: sh
**Step 4**: Modify source code files of style transfer sample from the cloned repository:<br>
pip3 install --user mxnet
pip3 install --user matplotlib
pip3 install --user scikit-image
1. Go to the `fast_mrf_cnn` subdirectory.
```sh
cd ./fast_mrf_cnn
```
2. Open the `symbol.py` file and modify the `decoder_symbol()` function. You should see the following code there:
```py
def decoder_symbol():
data = mx.sym.Variable('data')
data = mx.sym.Convolution(data=data, num_filter=256, kernel=(3,3), pad=(1,1), stride=(1, 1), name='deco_conv1')
```
Replace the code above with the following:<br>
```py
def decoder_symbol_with_vgg(vgg_symbol):
data = mx.sym.Convolution(data=vgg_symbol, num_filter=256, kernel=(3,3), pad=(1,1), stride=(1, 1), name='deco_conv1')
```
**Step 3**: Download the pre-trained `VGG19 model <https://github.com/dmlc/web-data/raw/master/mxnet/neural-style/model/vgg19.params>`__ and save it to the root directory of the cloned repository. The sample expects the model ``vgg19.params`` file to be in that directory.
3. Save and close the `symbol.py` file.
**Step 4**: Modify source code files of style transfer sample from the cloned repository:
4. Open and edit the `make_image.py` file. Go to the `__init__()` function in the `Maker` class:<br>
```py
decoder = symbol.decoder_symbol()
```
Modfiy it with the following code:<br>
```py
decoder = symbol.decoder_symbol_with_vgg(vgg_symbol)
```
1. Go to the ``fast_mrf_cnn`` subdirectory.
5. To join the pretrained weights with the decoder weights, make the following changes:
After the code lines for loading the decoder weights:<br>
```py
args = mx.nd.load('%s_decoder_args.nd'%model_prefix)
auxs = mx.nd.load('%s_decoder_auxs.nd'%model_prefix)
```
.. code-block:: sh
Add the following line:<br>
```py
arg_dict.update(args)
```
cd ./fast_mrf_cnn
6. Use `arg_dict` instead of `args` as a parameter of the `decoder.bind()` function. Find the line below:<br>
```py
self.deco_executor = decoder.bind(ctx=mx.gpu(), args=args, aux_states=auxs)
```
Replace it with the following:<br>
```py
self.deco_executor = decoder.bind(ctx=mx.cpu(), args=arg_dict, aux_states=auxs)
```
7. Add the following code to the end of the `generate()` function in the `Maker` class to save the result model as a `.json` file:<br>
```py
self.vgg_executor._symbol.save('{}-symbol.json'.format('vgg19'))
self.deco_executor._symbol.save('{}-symbol.json'.format('nst_vgg19'))
```
8. Save and close the `make_image.py` file.
**Step 5**: Follow the instructions from the `README.md` file in the `fast_mrf_cnn` directory of the cloned repository and run the sample with a decoder model.
For example, use the following code to run the sample with the pretrained decoder weights from the `models` folder and output shape:<br>
```py
import make_image
maker = make_image.Maker('models/13', (1024, 768))
maker.generate('output.jpg', '../images/tubingen.jpg')
```
The `models/13` string in the code above is composed of the following substrings:
* `models/` -- path to the folder that contains `.nd` files with pretrained styles weights.
* `13` -- prefix pointing to the default decoder for the repository, `13_decoder`.
2. Open the ``symbol.py`` file and modify the ``decoder_symbol()`` function. You should see the following code there:
> **NOTE**: If an error prompts with "No module named `cPickle`", try running the script from Step 5 in Python 2. After that return to Python 3 for the remaining steps.
.. code-block:: py
Any style can be selected from [collection of pretrained weights](https://pan.baidu.com/s/1skMHqYp). On the Chinese-language page, click the down arrow next to a size in megabytes. Then wait for an overlay box to appear, and click the blue button in it to download. The `generate()` function generates `nst_vgg19-symbol.json` and `vgg19-symbol.json` files for the specified shape. In the code, it is [1024 x 768] for a 4:3 ratio. You can specify another, for example, [224,224] for a square ratio.
def decoder_symbol():
data = mx.sym.Variable('data')
data = mx.sym.Convolution(data=data, num_filter=256, kernel=(3,3), pad=(1,1), stride=(1, 1), name='deco_conv1')
Replace the code above with the following:
.. code-block:: py
def decoder_symbol_with_vgg(vgg_symbol):
data = mx.sym.Convolution(data=vgg_symbol, num_filter=256, kernel=(3,3), pad=(1,1), stride=(1, 1), name='deco_conv1')
3. Save and close the ``symbol.py`` file.
4. Open and edit the ``make_image.py`` file. Go to the ``__init__()`` function in the ``Maker`` class:
.. code-block:: py
decoder = symbol.decoder_symbol()
Modify it with the following code:
.. code-block:: py
decoder = symbol.decoder_symbol_with_vgg(vgg_symbol)
5. To join the pre-trained weights with the decoder weights, make the following changes:
After the code lines for loading the decoder weights:
.. code-block:: py
args = mx.nd.load('%s_decoder_args.nd'%model_prefix)
auxs = mx.nd.load('%s_decoder_auxs.nd'%model_prefix)
Add the following line:
.. code-block:: py
arg_dict.update(args)
6. Use ``arg_dict`` instead of ``args`` as a parameter of the ``decoder.bind()`` function. Find the line below:
.. code-block:: py
self.deco_executor = decoder.bind(ctx=mx.gpu(), args=args, aux_states=auxs)
Replace it with the following:
.. code-block:: py
self.deco_executor = decoder.bind(ctx=mx.cpu(), args=arg_dict, aux_states=auxs)
7. Add the following code to the end of the ``generate()`` function in the ``Maker`` class to save the result model as a ``.json`` file:
.. code-block:: py
self.vgg_executor._symbol.save('{}-symbol.json'.format('vgg19'))
self.deco_executor._symbol.save('{}-symbol.json'.format('nst_vgg19'))
8. Save and close the ``make_image.py`` file.
**Step 5**: Follow the instructions from the ``README.md`` file in the ``fast_mrf_cnn`` directory of the cloned repository and run the sample with a decoder model.
For example, use the following code to run the sample with the pre-trained decoder weights from the ``models`` folder and output shape:
.. code-block:: py
import make_image
maker = make_image.Maker('models/13', (1024, 768))
maker.generate('output.jpg', '../images/tubingen.jpg')
The ``models/13`` string in the code above is composed of the following substrings:
* ``models/`` -- path to the folder that contains ``.nd`` files with pre-trained styles weights.
* ``13`` -- prefix pointing to the default decoder for the repository, ``13_decoder``.
.. note::
If an error prompts with "No module named ``cPickle``", try running the script from Step 5 in Python 2. After that return to Python 3 for the remaining steps.
Any style can be selected from `collection of pre-trained weights <https://pan.baidu.com/s/1skMHqYp>`__. On the Chinese-language page, click the down arrow next to a size in megabytes. Then wait for an overlay box to appear, and click the blue button in it to download. The ``generate()`` function generates ``nst_vgg19-symbol.json`` and ``vgg19-symbol.json`` files for the specified shape. In the code, it is ``[1024 x 768]`` for a 4:3 ratio. You can specify another, for example, ``[224,224]`` for a square ratio.
**Step 6**: Run the Model Optimizer to generate an Intermediate Representation (IR):
1. Create a new directory. For example:<br>
```sh
mkdir nst_model
```
2. Copy the initial and generated model files to the created directory. For example, to copy the pretrained decoder weights from the `models` folder to the `nst_model` directory, run the following commands:<br>
```sh
cp nst_vgg19-symbol.json nst_model
cp vgg19-symbol.json nst_model
cp ../vgg19.params nst_model/vgg19-0000.params
cp models/13_decoder_args.nd nst_model
cp models/13_decoder_auxs.nd nst_model
```
> **NOTE**: Make sure that all the `.params` and `.json` files are in the same directory as the `.nd` files. Otherwise, the conversion process fails.
1. Create a new directory. For example:
3. Run the Model Optimizer for Apache MXNet. Use the `--nd_prefix_name` option to specify the decoder prefix and `--input_shape` to specify input shapes in [N,C,W,H] order. For example:<br>
```sh
mo --input_symbol <path/to/nst_model>/nst_vgg19-symbol.json --framework mxnet --output_dir <path/to/output_dir> --input_shape [1,3,224,224] --nd_prefix_name 13_decoder --pretrained_model <path/to/nst_model>/vgg19-0000.params
```
4. The IR is generated (`.bin`, `.xml` and `.mapping` files) in the specified output directory, and ready to be consumed by the OpenVINO Runtime.
.. code-block:: sh
mkdir nst_model
2. Copy the initial and generated model files to the created directory. For example, to copy the pre-trained decoder weights from the ``models`` folder to the ``nst_model`` directory, run the following commands:
.. code-block:: sh
cp nst_vgg19-symbol.json nst_model
cp vgg19-symbol.json nst_model
cp ../vgg19.params nst_model/vgg19-0000.params
cp models/13_decoder_args.nd nst_model
cp models/13_decoder_auxs.nd nst_model
.. note::
Make sure that all the ``.params`` and ``.json`` files are in the same directory as the ``.nd`` files. Otherwise, the conversion process fails.
3. Run the Model Optimizer for Apache MXNet. Use the ``--nd_prefix_name`` option to specify the decoder prefix and ``--input_shape`` to specify input shapes in ``[N,C,W,H]`` order. For example:
.. code-block:: sh
mo --input_symbol <path/to/nst_model>/nst_vgg19-symbol.json --framework mxnet --output_dir <path/to/output_dir> --input_shape [1,3,224,224] --nd_prefix_name 13_decoder --pretrained_model <path/to/nst_model>/vgg19-0000.params
4. The IR is generated (``.bin``, ``.xml`` and ``.mapping`` files) in the specified output directory, and ready to be consumed by the OpenVINO Runtime.
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_Bert_ner.md
+46 -38
View File
@@ -1,58 +1,66 @@
# Converting a PyTorch BERT-NER Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_Bert_ner}
@sphinxdirective
The goal of this article is to present a step-by-step guide on how to convert PyTorch BERT-NER model to OpenVINO IR. First, you need to download the model and convert it to ONNX.
## Downloading and Converting the Model to ONNX
Downloading and Converting the Model to ONNX
############################################
To download a pretrained model or train the model yourself, refer
to the [instructions](https://github.com/kamalkraj/BERT-NER/blob/dev/README.md) in the
BERT-NER model repository. The model with configuration files is stored in the `out_base` directory.
to the `instructions <https://github.com/kamalkraj/BERT-NER/blob/dev/README.md>`__ in the
BERT-NER model repository. The model with configuration files is stored in the ``out_base`` directory.
To convert the model to ONNX format, create and run the following script in the root
directory of the model repository. If you download the pretrained model, you need
to download [bert.py](https://github.com/kamalkraj/BERT-NER/blob/dev/bert.py) to run the script.
The instructions were tested with the commit-SHA: `e5be564156f194f1becb0d82aeaf6e762d9eb9ed`.
to download `bert.py <https://github.com/kamalkraj/BERT-NER/blob/dev/bert.py>`__ to run the script.
The instructions were tested with the commit-SHA: ``e5be564156f194f1becb0d82aeaf6e762d9eb9ed``.
```python
import torch
.. code-block:: python
from bert import Ner
import torch
ner = Ner("out_base")
from bert import Ner
input_ids, input_mask, segment_ids, valid_positions = ner.preprocess('Steve went to Paris')
input_ids = torch.tensor([input_ids], dtype=torch.long, device=ner.device)
input_mask = torch.tensor([input_mask], dtype=torch.long, device=ner.device)
segment_ids = torch.tensor([segment_ids], dtype=torch.long, device=ner.device)
valid_ids = torch.tensor([valid_positions], dtype=torch.long, device=ner.device)
ner = Ner("out_base")
ner_model, tknizr, model_config = ner.load_model("out_base")
input_ids, input_mask, segment_ids, valid_positions = ner.preprocess('Steve went to Paris')
input_ids = torch.tensor([input_ids], dtype=torch.long, device=ner.device)
input_mask = torch.tensor([input_mask], dtype=torch.long, device=ner.device)
segment_ids = torch.tensor([segment_ids], dtype=torch.long, device=ner.device)
valid_ids = torch.tensor([valid_positions], dtype=torch.long, device=ner.device)
with torch.no_grad():
logits = ner_model(input_ids, segment_ids, input_mask, valid_ids)
torch.onnx.export(ner_model,
(input_ids, segment_ids, input_mask, valid_ids),
"bert-ner.onnx",
input_names=['input_ids', 'segment_ids', 'input_mask', 'valid_ids'],
output_names=['output'],
dynamic_axes={
"input_ids": {0: "batch_size"},
"segment_ids": {0: "batch_size"},
"input_mask": {0: "batch_size"},
"valid_ids": {0: "batch_size"},
"output": {0: "output"}
},
opset_version=11,
)
```
ner_model, tknizr, model_config = ner.load_model("out_base")
The script generates ONNX model file `bert-ner.onnx`.
with torch.no_grad():
logits = ner_model(input_ids, segment_ids, input_mask, valid_ids)
torch.onnx.export(ner_model,
(input_ids, segment_ids, input_mask, valid_ids),
"bert-ner.onnx",
input_names=['input_ids', 'segment_ids', 'input_mask', 'valid_ids'],
output_names=['output'],
dynamic_axes={
"input_ids": {0: "batch_size"},
"segment_ids": {0: "batch_size"},
"input_mask": {0: "batch_size"},
"valid_ids": {0: "batch_size"},
"output": {0: "output"}
},
opset_version=11,
)
## Converting an ONNX BERT-NER model to IR
```bash
mo --input_model bert-ner.onnx --input "input_mask[1,128],segment_ids[1,128],input_ids[1,128]"
```
The script generates ONNX model file ``bert-ner.onnx``.
where `1` is `batch_size` and `128` is `sequence_length`.
Converting an ONNX BERT-NER model to IR
#######################################
.. code-block:: bash
mo --input_model bert-ner.onnx --input "input_mask[1,128],segment_ids[1,128],input_ids[1,128]"
where ``1`` is ``batch_size`` and ``128`` is ``sequence_length``.
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_Cascade_RCNN_res101.md
+28 -17
View File
@@ -1,30 +1,41 @@
# Converting a PyTorch Cascade RCNN R-101 Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_Cascade_RCNN_res101}
@sphinxdirective
The goal of this article is to present a step-by-step guide on how to convert a PyTorch Cascade RCNN R-101 model to OpenVINO IR. First, you need to download the model and convert it to ONNX.
## Downloading and Converting Model to ONNX
Downloading and Converting Model to ONNX
########################################
* Clone the [repository](https://github.com/open-mmlab/mmdetection):
* Clone the `repository <https://github.com/open-mmlab/mmdetection>`__ :
```bash
git clone https://github.com/open-mmlab/mmdetection
cd mmdetection
```
.. code-block:: sh
> **NOTE**: To set up an environment, refer to the [instructions](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md#installation).
git clone https://github.com/open-mmlab/mmdetection
cd mmdetection
* Download the pretrained [model](https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco/cascade_rcnn_r101_fpn_1x_coco_20200317-0b6a2fbf.pth). The model is also available [here](https://github.com/open-mmlab/mmdetection/blob/master/configs/cascade_rcnn/README.md).
* To convert the model to ONNX format, use this [script](https://github.com/open-mmlab/mmdetection/blob/master/tools/deployment/pytorch2onnx.py).
.. note::
```bash
python3 tools/deployment/pytorch2onnx.py configs/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco.py cascade_rcnn_r101_fpn_1x_coco_20200317-0b6a2fbf.pth --output-file cascade_rcnn_r101_fpn_1x_coco.onnx
```
To set up an environment, refer to the `instructions <https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md#installation>`__.
The script generates ONNX model file `cascade_rcnn_r101_fpn_1x_coco.onnx` in the directory `tools/deployment/`. If required, specify the model name or output directory, using `--output-file <path-to-dir>/<model-name>.onnx`.
* Download the pre-trained `model <https://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco/cascade_rcnn_r101_fpn_1x_coco_20200317-0b6a2fbf.pth>`__. The model is also available `here <https://github.com/open-mmlab/mmdetection/blob/master/configs/cascade_rcnn/README.md>`__.
## Converting an ONNX Cascade RCNN R-101 Model to OpenVINO IR
* To convert the model to ONNX format, use this `script <https://github.com/open-mmlab/mmdetection/blob/master/tools/deployment/pytorch2onnx.py>`__.
```bash
mo --input_model cascade_rcnn_r101_fpn_1x_coco.onnx --mean_values [123.675,116.28,103.53] --scale_values [58.395,57.12,57.375]
```
.. code-block:: sh
python3 tools/deployment/pytorch2onnx.py configs/cascade_rcnn/cascade_rcnn_r101_fpn_1x_coco.py cascade_rcnn_r101_fpn_1x_coco_20200317-0b6a2fbf.pth --output-file cascade_rcnn_r101_fpn_1x_coco.onnx
The script generates ONNX model file ``cascade_rcnn_r101_fpn_1x_coco.onnx`` in the directory ``tools/deployment/``. If required, specify the model name or output directory, using ``--output-file <path-to-dir>/<model-name>.onnx``.
Converting an ONNX Cascade RCNN R-101 Model to OpenVINO IR
##########################################################
.. code-block:: sh
mo --input_model cascade_rcnn_r101_fpn_1x_coco.onnx --mean_values [123.675,116.28,103.53] --scale_values [58.395,57.12,57.375]
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_F3Net.md
+33 -21
View File
@@ -1,33 +1,45 @@
# Converting a PyTorch F3Net Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_F3Net}
[F3Net](https://github.com/weijun88/F3Net): Fusion, Feedback and Focus for Salient Object Detection
@sphinxdirective
## Cloning the F3Net Repository
`F3Net <https://github.com/weijun88/F3Net>`__ : Fusion, Feedback and Focus for Salient Object Detection
Cloning the F3Net Repository
############################
To clone the repository, run the following command:
```sh
git clone http://github.com/weijun88/F3Net.git
```
.. code-block:: sh
## Downloading and Converting the Model to ONNX
git clone http://github.com/weijun88/F3Net.git
Downloading and Converting the Model to ONNX
############################################
To download the pretrained model or train the model yourself, refer to the
[instructions](https://github.com/weijun88/F3Net/blob/master/README.md) in the F3Net model repository. First, convert the model to ONNX format. Create and run the following Python script in the `src` directory of the model repository:
```python
import torch
from dataset import Config
from net import F3Net
`instructions <https://github.com/weijun88/F3Net/blob/master/README.md>`__ in the F3Net model repository. First, convert the model to ONNX format. Create and run the following Python script in the ``src`` directory of the model repository:
cfg = Config(mode='test', snapshot=<path_to_checkpoint_dir>)
net = F3Net(cfg)
image = torch.zeros([1, 3, 352, 352])
torch.onnx.export(net, image, 'f3net.onnx', export_params=True, do_constant_folding=True, opset_version=11)
```
The script generates the ONNX model file `f3net.onnx`. The model conversion was tested with the commit-SHA: `eecace3adf1e8946b571a4f4397681252f9dc1b8`.
.. code-block:: python
## Converting an ONNX F3Net Model to IR
import torch
from dataset import Config
from net import F3Net
```sh
mo --input_model <MODEL_DIR>/f3net.onnx
```
cfg = Config(mode='test', snapshot=<path_to_checkpoint_dir>)
net = F3Net(cfg)
image = torch.zeros([1, 3, 352, 352])
torch.onnx.export(net, image, 'f3net.onnx', export_params=True, do_constant_folding=True, opset_version=11)
The script generates the ONNX model file ``f3net.onnx``. The model conversion was tested with the commit-SHA: ``eecace3adf1e8946b571a4f4397681252f9dc1b8``.
Converting an ONNX F3Net Model to IR
####################################
.. code-block:: sh
mo --input_model <MODEL_DIR>/f3net.onnx
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_QuartzNet.md
+36 -25
View File
@@ -1,40 +1,51 @@
# Converting a PyTorch QuartzNet Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_QuartzNet}
[NeMo project](https://github.com/NVIDIA/NeMo) provides the QuartzNet model.
@sphinxdirective
## Downloading the Pretrained QuartzNet Model
`NeMo project <https://github.com/NVIDIA/NeMo>`__ provides the QuartzNet model.
To download the pretrained model, refer to the [NeMo Speech Models Catalog](https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels).
Downloading the Pre-trained QuartzNet Model
###########################################
To download the pre-trained model, refer to the `NeMo Speech Models Catalog <https://ngc.nvidia.com/catalog/models/nvidia:nemospeechmodels>`__.
Here are the instructions on how to obtain QuartzNet in ONNX format.
1. Install the NeMo toolkit, using the [instructions](https://github.com/NVIDIA/NeMo/tree/main#installation).
1. Install the NeMo toolkit, using the `instructions <https://github.com/NVIDIA/NeMo/tree/main#installation>`__.
2. Run the following code:
```python
import nemo
import nemo.collections.asr as nemo_asr
.. code-block:: python
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
# Export QuartzNet model to ONNX format
quartznet.decoder.export('decoder_qn.onnx')
quartznet.encoder.export('encoder_qn.onnx')
quartznet.export('qn.onnx')
```
import nemo
import nemo.collections.asr as nemo_asr
This code produces 3 ONNX model files: `encoder_qn.onnx`, `decoder_qn.onnx`, `qn.onnx`.
They are `decoder`, `encoder`, and a combined `decoder(encoder(x))` models, respectively.
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
# Export QuartzNet model to ONNX format
quartznet.decoder.export('decoder_qn.onnx')
quartznet.encoder.export('encoder_qn.onnx')
quartznet.export('qn.onnx')
## Converting an ONNX QuartzNet model to IR
This code produces 3 ONNX model files: ``encoder_qn.onnx``, ``decoder_qn.onnx``, ``qn.onnx``.
They are ``decoder``, ``encoder``, and a combined ``decoder(encoder(x))`` models, respectively.
Converting an ONNX QuartzNet model to IR
########################################
If using a combined model:
```sh
mo --input_model <MODEL_DIR>/qt.onnx --input_shape [B,64,X]
```
If using separate models:
```sh
mo --input_model <MODEL_DIR>/encoder_qt.onnx --input_shape [B,64,X]
mo --input_model <MODEL_DIR>/decoder_qt.onnx --input_shape [B,1024,Y]
```
Where shape is determined by the audio file Mel-Spectrogram length: B - batch dimension, X - dimension based on the input length, Y - determined by encoder output, usually `X / 2`.
.. code-block:: sh
mo --input_model <MODEL_DIR>/qt.onnx --input_shape [B,64,X]
If using separate models:
.. code-block:: sh
mo --input_model <MODEL_DIR>/encoder_qt.onnx --input_shape [B,64,X]
mo --input_model <MODEL_DIR>/decoder_qt.onnx --input_shape [B,1024,Y]
Where shape is determined by the audio file Mel-Spectrogram length: ``B`` - batch dimension, ``X`` - dimension based on the input length, ``Y`` - determined by encoder output, usually ``X / 2``.
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_RCAN.md
+28 -19
View File
@@ -1,30 +1,39 @@
# Converting a PyTorch RCAN Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_RCAN}
[RCAN](https://github.com/yulunzhang/RCAN): Image Super-Resolution Using Very Deep Residual Channel Attention Networks
@sphinxdirective
## Downloading and Converting the Model to ONNX
`RCAN <https://github.com/yulunzhang/RCAN>`__ : Image Super-Resolution Using Very Deep Residual Channel Attention Networks
To download the pretrained model or train the model yourself, refer to the [instruction](https://github.com/yulunzhang/RCAN/blob/master/README.md) in the RCAN model repository. First, convert the model to ONNX format. Create and run the script with the following content in the root
Downloading and Converting the Model to ONNX
############################################
To download the pre-trained model or train the model yourself, refer to the `instruction <https://github.com/yulunzhang/RCAN/blob/master/README.md>`__ in the RCAN model repository. First, convert the model to ONNX format. Create and run the script with the following content in the root
directory of the model repository:
```python
from argparse import Namespace
.. code-block:: python
import torch
from argparse import Namespace
from RCAN_TestCode.code.model.rcan import RCAN
import torch
config = Namespace(n_feats=64, n_resblocks=4, n_resgroups=2, reduction=16, scale=[2], data_train='DIV2K', res_scale=1,
n_colors=3, rgb_range=255)
net = RCAN(config)
net.eval()
dummy_input = torch.randn(1, 3, 360, 640)
torch.onnx.export(net, dummy_input, 'RCAN.onnx')
```
The script generates the ONNX model file `RCAN.onnx`. More information about model parameters (`n_resblocks`, `n_resgroups`, and others) and their different values can be found in the model repository. The model conversion was tested with the commit-SHA: `3339ebc59519c3bb2b5719b87dd36515ec7f3ba7`.
from RCAN_TestCode.code.model.rcan import RCAN
## Converting an ONNX RCAN Model to IR
config = Namespace(n_feats=64, n_resblocks=4, n_resgroups=2, reduction=16, scale=[2], data_train='DIV2K', res_scale=1,
n_colors=3, rgb_range=255)
net = RCAN(config)
net.eval()
dummy_input = torch.randn(1, 3, 360, 640)
torch.onnx.export(net, dummy_input, 'RCAN.onnx')
```sh
mo --input_model RCAN.onnx
```
The script generates the ONNX model file ``RCAN.onnx``. More information about model parameters (``n_resblocks``, ``n_resgroups``, and others) and their different values can be found in the model repository. The model conversion was tested with the commit-SHA: ``3339ebc59519c3bb2b5719b87dd36515ec7f3ba7``.
Converting an ONNX RCAN Model to IR
###################################
.. code-block:: sh
mo --input_model RCAN.onnx
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_RNNT.md
+95 -74
View File
@@ -1,106 +1,127 @@
# Converting a PyTorch RNN-T Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_RNNT}
This guide covers conversion of RNN-T model from [MLCommons](https://github.com/mlcommons) repository. Follow
@sphinxdirective
This guide covers conversion of RNN-T model from `MLCommons <https://github.com/mlcommons>`__ repository. Follow
the instructions below to export a PyTorch model into ONNX, before converting it to IR:
**Step 1**. Clone RNN-T PyTorch implementation from MLCommons repository (revision r1.0). Make a shallow clone to pull
only RNN-T model without full repository. If you already have a full repository, skip this and go to **Step 2**:
```bash
git clone -b r1.0 -n https://github.com/mlcommons/inference rnnt_for_openvino --depth 1
cd rnnt_for_openvino
git checkout HEAD speech_recognition/rnnt
```
.. code-block:: sh
git clone -b r1.0 -n https://github.com/mlcommons/inference rnnt_for_openvino --depth 1
cd rnnt_for_openvino
git checkout HEAD speech_recognition/rnnt
**Step 2**. If you already have a full clone of MLCommons inference repository, create a folder for
pretrained PyTorch model, where conversion into IR will take place. You will also need to specify the path to
your full clone at **Step 5**. Skip this step if you have a shallow clone.
```bash
mkdir rnnt_for_openvino
cd rnnt_for_openvino
```
.. code-block:: sh
**Step 3**. Download pretrained weights for PyTorch implementation from [here](https://zenodo.org/record/3662521#.YG21DugzZaQ).
For UNIX-like systems, you can use `wget`:
```bash
wget https://zenodo.org/record/3662521/files/DistributedDataParallel_1576581068.9962234-epoch-100.pt
```
The link was taken from `setup.sh` in the `speech_recoginitin/rnnt` subfolder. You will get exactly the same weights as
if you were following the [guide](https://github.com/mlcommons/inference/tree/master/speech_recognition/rnnt).
mkdir rnnt_for_openvino
cd rnnt_for_openvino
**Step 3**. Download pre-trained weights for PyTorch implementation from `here <https://zenodo.org/record/3662521#.YG21DugzZaQ>`__.
For UNIX-like systems, you can use ``wget``:
.. code-block:: sh
wget https://zenodo.org/record/3662521/files/DistributedDataParallel_1576581068.9962234-epoch-100.pt
The link was taken from ``setup.sh`` in the ``speech_recoginitin/rnnt`` subfolder. You will get exactly the same weights as
if you were following the `guide <https://github.com/mlcommons/inference/tree/master/speech_recognition/rnnt>`__.
**Step 4**. Install required Python packages:
```bash
pip3 install torch toml
```
.. code-block:: sh
pip3 install torch toml
**Step 5**. Export RNN-T model into ONNX, using the script below. Copy the code below into a file named
`export_rnnt_to_onnx.py` and run it in the current directory `rnnt_for_openvino`:
``export_rnnt_to_onnx.py`` and run it in the current directory ``rnnt_for_openvino``:
> **NOTE**: If you already have a full clone of MLCommons inference repository, you need to
> specify the `mlcommons_inference_path` variable.
.. note::
```python
import toml
import torch
import sys
If you already have a full clone of MLCommons inference repository, you need
to specify the ``mlcommons_inference_path`` variable.
.. code-block:: python
import toml
import torch
import sys
def load_and_migrate_checkpoint(ckpt_path):
checkpoint = torch.load(ckpt_path, map_location="cpu")
migrated_state_dict = {}
for key, value in checkpoint['state_dict'].items():
key = key.replace("joint_net", "joint.net")
migrated_state_dict[key] = value
del migrated_state_dict["audio_preprocessor.featurizer.fb"]
del migrated_state_dict["audio_preprocessor.featurizer.window"]
return migrated_state_dict
def load_and_migrate_checkpoint(ckpt_path):
checkpoint = torch.load(ckpt_path, map_location="cpu")
migrated_state_dict = {}
for key, value in checkpoint['state_dict'].items():
key = key.replace("joint_net", "joint.net")
migrated_state_dict[key] = value
del migrated_state_dict["audio_preprocessor.featurizer.fb"]
del migrated_state_dict["audio_preprocessor.featurizer.window"]
return migrated_state_dict
mlcommons_inference_path = './' # specify relative path for MLCommons inferene
checkpoint_path = 'DistributedDataParallel_1576581068.9962234-epoch-100.pt'
config_toml = 'speech_recognition/rnnt/pytorch/configs/rnnt.toml'
config = toml.load(config_toml)
rnnt_vocab = config['labels']['labels']
sys.path.insert(0, mlcommons_inference_path + 'speech_recognition/rnnt/pytorch')
mlcommons_inference_path = './' # specify relative path for MLCommons inferene
checkpoint_path = 'DistributedDataParallel_1576581068.9962234-epoch-100.pt'
config_toml = 'speech_recognition/rnnt/pytorch/configs/rnnt.toml'
config = toml.load(config_toml)
rnnt_vocab = config['labels']['labels']
sys.path.insert(0, mlcommons_inference_path + 'speech_recognition/rnnt/pytorch')
from model_separable_rnnt import RNNT
from model_separable_rnnt import RNNT
model = RNNT(config['rnnt'], len(rnnt_vocab) + 1, feature_config=config['input_eval'])
model.load_state_dict(load_and_migrate_checkpoint(checkpoint_path))
model = RNNT(config['rnnt'], len(rnnt_vocab) + 1, feature_config=config['input_eval'])
model.load_state_dict(load_and_migrate_checkpoint(checkpoint_path))
seq_length, batch_size, feature_length = 157, 1, 240
inp = torch.randn([seq_length, batch_size, feature_length])
feature_length = torch.LongTensor([seq_length])
x_padded, x_lens = model.encoder(inp, feature_length)
torch.onnx.export(model.encoder, (inp, feature_length), "rnnt_encoder.onnx", opset_version=12,
input_names=['input', 'feature_length'], output_names=['x_padded', 'x_lens'],
dynamic_axes={'input': {0: 'seq_len', 1: 'batch'}})
seq_length, batch_size, feature_length = 157, 1, 240
inp = torch.randn([seq_length, batch_size, feature_length])
feature_length = torch.LongTensor([seq_length])
x_padded, x_lens = model.encoder(inp, feature_length)
torch.onnx.export(model.encoder, (inp, feature_length), "rnnt_encoder.onnx", opset_version=12,
input_names=['input', 'feature_length'], output_names=['x_padded', 'x_lens'],
dynamic_axes={'input': {0: 'seq_len', 1: 'batch'}})
symbol = torch.LongTensor([[20]])
hidden = torch.randn([2, batch_size, 320]), torch.randn([2, batch_size, 320])
g, hidden = model.prediction.forward(symbol, hidden)
torch.onnx.export(model.prediction, (symbol, hidden), "rnnt_prediction.onnx", opset_version=12,
input_names=['symbol', 'hidden_in_1', 'hidden_in_2'],
output_names=['g', 'hidden_out_1', 'hidden_out_2'],
dynamic_axes={'symbol': {0: 'batch'}, 'hidden_in_1': {1: 'batch'}, 'hidden_in_2': {1: 'batch'}})
symbol = torch.LongTensor([[20]])
hidden = torch.randn([2, batch_size, 320]), torch.randn([2, batch_size, 320])
g, hidden = model.prediction.forward(symbol, hidden)
torch.onnx.export(model.prediction, (symbol, hidden), "rnnt_prediction.onnx", opset_version=12,
input_names=['symbol', 'hidden_in_1', 'hidden_in_2'],
output_names=['g', 'hidden_out_1', 'hidden_out_2'],
dynamic_axes={'symbol': {0: 'batch'}, 'hidden_in_1': {1: 'batch'}, 'hidden_in_2': {1: 'batch'}})
f = torch.randn([batch_size, 1, 1024])
model.joint.forward(f, g)
torch.onnx.export(model.joint, (f, g), "rnnt_joint.onnx", opset_version=12,
input_names=['0', '1'], output_names=['result'], dynamic_axes={'0': {0: 'batch'}, '1': {0: 'batch'}})
```
f = torch.randn([batch_size, 1, 1024])
model.joint.forward(f, g)
torch.onnx.export(model.joint, (f, g), "rnnt_joint.onnx", opset_version=12,
input_names=['0', '1'], output_names=['result'], dynamic_axes={'0': {0: 'batch'}, '1': {0: 'batch'}})
```bash
python3 export_rnnt_to_onnx.py
```
After completing this step, the files `rnnt_encoder.onnx`, `rnnt_prediction.onnx`, and `rnnt_joint.onnx` will be saved in the current directory.
.. code-block:: sh
python3 export_rnnt_to_onnx.py
After completing this step, the files ``rnnt_encoder.onnx``, ``rnnt_prediction.onnx``, and ``rnnt_joint.onnx`` will be saved in the current directory.
**Step 6**. Run the conversion commands:
```sh
mo --input_model rnnt_encoder.onnx --input "input[157,1,240],feature_length->157"
mo --input_model rnnt_prediction.onnx --input "symbol[1,1],hidden_in_1[2,1,320],hidden_in_2[2,1,320]"
mo --input_model rnnt_joint.onnx --input "0[1,1,1024],1[1,1,320]"
```
> **NOTE**: The hardcoded value for sequence length = 157 was taken from the MLCommons, but conversion to IR preserves network [reshapeability](@ref openvino_docs_OV_UG_ShapeInference). Therefore, input shapes can be changed manually to any value during either conversion or inference.
.. code-block:: sh
mo --input_model rnnt_encoder.onnx --input "input[157,1,240],feature_length->157"
mo --input_model rnnt_prediction.onnx --input "symbol[1,1],hidden_in_1[2,1,320],hidden_in_2[2,1,320]"
mo --input_model rnnt_joint.onnx --input "0[1,1,1024],1[1,1,320]"
.. note::
The hardcoded value for sequence length = 157 was taken from the MLCommons, but conversion to IR preserves network :doc:`reshapeability <openvino_docs_OV_UG_ShapeInference>`. Therefore, input shapes can be changed manually to any value during either conversion or inference.
@endsphinxdirective
docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_YOLACT.md
+157 -132
View File
@@ -1,187 +1,212 @@
# Converting a PyTorch YOLACT Model {#openvino_docs_MO_DG_prepare_model_convert_model_pytorch_specific_Convert_YOLACT}
@sphinxdirective
You Only Look At CoefficienTs (YOLACT) is a simple, fully convolutional model for real-time instance segmentation.
The PyTorch implementation is publicly available in [this GitHub repository](https://github.com/dbolya/yolact).
The PyTorch implementation is publicly available in `this GitHub repository <https://github.com/dbolya/yolact>`__.
The YOLACT++ model is not supported, because it uses deformable convolutional layers that cannot be represented in ONNX format.
## Creating a Patch File <a name="patch-file-yolact"></a>
.. _patch-file-yolact:
Creating a Patch File
#####################
Before converting the model, create a patch file for the repository.
The patch modifies the framework code by adding a special command-line argument to the framework options. The argument enables inference graph dumping:
1. Go to a writable directory and create a `YOLACT_onnx_export.patch` file.
1. Go to a writable directory and create a ``YOLACT_onnx_export.patch`` file.
2. Copy the following diff code to the file:
```git
From 76deb67d4f09f29feda1a633358caa18335d9e9f Mon Sep 17 00:00:00 2001
From: "OpenVINO" <openvino@intel.com>
Date: Fri, 12 Mar 2021 00:27:35 +0300
Subject: [PATCH] Add export to ONNX
---
eval.py | 5 ++++-
utils/augmentations.py | 7 +++++--
yolact.py | 29 +++++++++++++++++++----------
3 files changed, 28 insertions(+), 13 deletions(-)
.. code-block:: console
diff --git a/eval.py b/eval.py
index 547bc0a..bde0680 100644
--- a/eval.py
+++ b/eval.py
@@ -593,9 +593,12 @@ def badhash(x):
return x
From 76deb67d4f09f29feda1a633358caa18335d9e9f Mon Sep 17 00:00:00 2001
From: "OpenVINO" <openvino@intel.com>
Date: Fri, 12 Mar 2021 00:27:35 +0300
Subject: [PATCH] Add export to ONNX
def evalimage(net:Yolact, path:str, save_path:str=None):
- frame = torch.from_numpy(cv2.imread(path)).cuda().float()
+ frame = torch.from_numpy(cv2.imread(path)).float()
+ if torch.cuda.is_available():
+ frame = frame.cuda()
batch = FastBaseTransform()(frame.unsqueeze(0))
preds = net(batch)
+ torch.onnx.export(net, batch, "yolact.onnx", opset_version=11)
---
eval.py | 5 ++++-
utils/augmentations.py | 7 +++++--
yolact.py | 29 +++++++++++++++++++----------
3 files changed, 28 insertions(+), 13 deletions(-)
img_numpy = prep_display(preds, frame, None, None, undo_transform=False)
diff --git a/eval.py b/eval.py
index 547bc0a..bde0680 100644
--- a/eval.py
+++ b/eval.py
@@ -593,9 +593,12 @@ def badhash(x):
return x
diff --git a/utils/augmentations.py b/utils/augmentations.py
index cc7a73a..2420603 100644
--- a/utils/augmentations.py
+++ b/utils/augmentations.py
@@ -623,8 +623,11 @@ class FastBaseTransform(torch.nn.Module):
def __init__(self):
super().__init__()
def evalimage(net:Yolact, path:str, save_path:str=None):
- frame = torch.from_numpy(cv2.imread(path)).cuda().float()
+ frame = torch.from_numpy(cv2.imread(path)).float()
+ if torch.cuda.is_available():
+ frame = frame.cuda()
batch = FastBaseTransform()(frame.unsqueeze(0))
preds = net(batch)
+ torch.onnx.export(net, batch, "yolact.onnx", opset_version=11)
- self.mean = torch.Tensor(MEANS).float().cuda()[None, :, None, None]
- self.std = torch.Tensor( STD ).float().cuda()[None, :, None, None]
+ self.mean = torch.Tensor(MEANS).float()[None, :, None, None]
+ self.std = torch.Tensor( STD ).float()[None, :, None, None]
+ if torch.cuda.is_available():
+ self.mean.cuda()
+ self.std.cuda()
self.transform = cfg.backbone.transform
img_numpy = prep_display(preds, frame, None, None, undo_transform=False)
def forward(self, img):
diff --git a/yolact.py b/yolact.py
index d83703b..f8c787c 100644
--- a/yolact.py
+++ b/yolact.py
@@ -17,19 +17,22 @@ import torch.backends.cudnn as cudnn
from utils import timer
from utils.functions import MovingAverage, make_net
diff --git a/utils/augmentations.py b/utils/augmentations.py
index cc7a73a..2420603 100644
--- a/utils/augmentations.py
+++ b/utils/augmentations.py
@@ -623,8 +623,11 @@ class FastBaseTransform(torch.nn.Module):
def __init__(self):
super().__init__()
-# This is required for Pytorch 1.0.1 on Windows to initialize Cuda on some driver versions.
-# See the bug report here: https://github.com/pytorch/pytorch/issues/17108
-torch.cuda.current_device()
-
-# As of March 10, 2019, Pytorch DataParallel still doesn't support JIT Script Modules
-use_jit = torch.cuda.device_count() <= 1
-if not use_jit:
- print('Multiple GPUs detected! Turning off JIT.')
+use_jit = False
- self.mean = torch.Tensor(MEANS).float().cuda()[None, :, None, None]
- self.std = torch.Tensor( STD ).float().cuda()[None, :, None, None]
+ self.mean = torch.Tensor(MEANS).float()[None, :, None, None]
+ self.std = torch.Tensor( STD ).float()[None, :, None, None]
+ if torch.cuda.is_available():
+ self.mean.cuda()
+ self.std.cuda()
self.transform = cfg.backbone.transform
ScriptModuleWrapper = torch.jit.ScriptModule if use_jit else nn.Module
script_method_wrapper = torch.jit.script_method if use_jit else lambda fn, _rcn=None: fn
def forward(self, img):
diff --git a/yolact.py b/yolact.py
index d83703b..f8c787c 100644
--- a/yolact.py
+++ b/yolact.py
@@ -17,19 +17,22 @@ import torch.backends.cudnn as cudnn
from utils import timer
from utils.functions import MovingAverage, make_net
-# This is required for Pytorch 1.0.1 on Windows to initialize Cuda on some driver versions.
-# See the bug report here: https://github.com/pytorch/pytorch/issues/17108
-torch.cuda.current_device()
-
-# As of March 10, 2019, Pytorch DataParallel still doesn't support JIT Script Modules
-use_jit = torch.cuda.device_count() <= 1
-if not use_jit:
- print('Multiple GPUs detected! Turning off JIT.')
+use_jit = False
ScriptModuleWrapper = torch.jit.ScriptModule if use_jit else nn.Module
script_method_wrapper = torch.jit.script_method if use_jit else lambda fn, _rcn=None: fn
+def decode(loc, priors):
+ variances = [0.1, 0.2]
+ boxes = torch.cat((priors[:, :2] + loc[:, :, :2] * variances[0] * priors[:, 2:], priors[:, 2:] * torch.exp(loc[:, :, 2:] * variances[1])), 2)
+
+ boxes_result1 = boxes[:, :, :2] - boxes[:, :, 2:] / 2
+ boxes_result2 = boxes[:, :, 2:] + boxes_result1
+ boxes_result = torch.cat((boxes_result1, boxes_result2), 2)
+
+ return boxes_result
+
+def decode(loc, priors):
+ variances = [0.1, 0.2]
+ boxes = torch.cat((priors[:, :2] + loc[:, :, :2] * variances[0] * priors[:, 2:], priors[:, 2:] * torch.exp(loc[:, :, 2:] * variances[1])), 2)
+
+ boxes_result1 = boxes[:, :, :2] - boxes[:, :, 2:] / 2
+ boxes_result2 = boxes[:, :, 2:] + boxes_result1
+ boxes_result = torch.cat((boxes_result1, boxes_result2), 2)
+
+ return boxes_result
+
class Concat(nn.Module):
def __init__(self, nets, extra_params):
@@ -476,7 +479,10 @@ class Yolact(nn.Module):
class Concat(nn.Module):
def __init__(self, nets, extra_params):
@@ -476,7 +479,10 @@ class Yolact(nn.Module):
def load_weights(self, path):
""" Loads weights from a compressed save file. """
- state_dict = torch.load(path)
+ if torch.cuda.is_available():
+ state_dict = torch.load(path)
+ else:
+ state_dict = torch.load(path, map_location=torch.device('cpu'))
def load_weights(self, path):
""" Loads weights from a compressed save file. """
- state_dict = torch.load(path)
+ if torch.cuda.is_available():
+ state_dict = torch.load(path)
+ else:
+ state_dict = torch.load(path, map_location=torch.device('cpu'))
# For backward compatability, remove these (the new variable is called layers)
for key in list(state_dict.keys()):
@@ -673,8 +679,11 @@ class Yolact(nn.Module):
else:
pred_outs['conf'] = F.softmax(pred_outs['conf'], -1)
# For backward compatability, remove these (the new variable is called layers)
for key in list(state_dict.keys()):
@@ -673,8 +679,11 @@ class Yolact(nn.Module):
else:
pred_outs['conf'] = F.softmax(pred_outs['conf'], -1)
- return self.detect(pred_outs, self)
+ pred_outs['boxes'] = decode(pred_outs['loc'], pred_outs['priors']) # decode output boxes
- return self.detect(pred_outs, self)
+ pred_outs['boxes'] = decode(pred_outs['loc'], pred_outs['priors']) # decode output boxes
+ pred_outs.pop('priors') # remove unused in postprocessing layers
+ pred_outs.pop('loc') # remove unused in postprocessing layers
+ return pred_outs
+ pred_outs.pop('priors') # remove unused in postprocessing layers
+ pred_outs.pop('loc') # remove unused in postprocessing layers
+ return pred_outs
--
```
--
3. Save and close the file.
## Converting a YOLACT Model to the OpenVINO IR format
Converting a YOLACT Model to the OpenVINO IR format
###################################################
**Step 1**. Clone the GitHub repository and check out the commit:
1. Clone the YOLACT repository:
```sh
git clone https://github.com/dbolya/yolact
```
.. code-block:: sh
git clone https://github.com/dbolya/yolact
2. Check out the necessary commit:
```sh
git checkout 57b8f2d95e62e2e649b382f516ab41f949b57239
```
3. Set up the environment as described in `README.md`.
.. code-block:: sh
**Step 2**. Download a pretrained model from the list attached in the `Evaluation` section of `README.md` document, for example `yolact_base_54_800000.pth`.
git checkout 57b8f2d95e62e2e649b382f516ab41f949b57239
3. Set up the environment as described in ``README.md``.
**Step 2**. Download a pre-trained model from the list attached in the ``Evaluation`` section of ``README.md`` document, for example ``yolact_base_54_800000.pth``.
**Step 3**. Export the model to ONNX format.
1. Apply the `YOLACT_onnx_export.patch` patch to the repository. Refer to the <a href="#patch-file-yolact">Create a Patch File</a> instructions if you do not have it:
```sh
git apply /path/to/patch/YOLACT_onnx_export.patch
```
1. Apply the `YOLACT_onnx_export.patch` patch to the repository. Refer to the :ref:`Create a Patch File <patch-file-yolact>` instructions if you do not have it:
.. code-block:: sh
git apply /path/to/patch/YOLACT_onnx_export.patch
2. Evaluate the YOLACT model to export it to ONNX format:
```sh
python3 eval.py \
--trained_model=/path/to/yolact_base_54_800000.pth \
--score_threshold=0.3 \
--top_k=10 \
--image=/path/to/image.jpg \
--cuda=False
```
.. code-block:: sh
3. The script may fail, but you should get `yolact.onnx` file.
python3 eval.py \
--trained_model=/path/to/yolact_base_54_800000.pth \
--score_threshold=0.3 \
--top_k=10 \
--image=/path/to/image.jpg \
--cuda=False
3. The script may fail, but you should get ``yolact.onnx`` file.
**Step 4**. Convert the model to the IR:
```sh
mo --input_model /path/to/yolact.onnx
```
.. code-block:: sh
mo --input_model /path/to/yolact.onnx
**Step 4**. Embed input preprocessing into the IR:
To get performance gain by offloading to the OpenVINO application of mean/scale values and RGB->BGR conversion, use the following options of the Model Optimizer (MO):
* If the backbone of the model is Resnet50-FPN or Resnet101-FPN, use the following MO command line:
```sh
mo \
--input_model /path/to/yolact.onnx \
--reverse_input_channels \
--mean_values "[123.68, 116.78, 103.94]" \
--scale_values "[58.40, 57.12, 57.38]"
```
.. code-block:: sh
mo \
--input_model /path/to/yolact.onnx \
--reverse_input_channels \
--mean_values "[123.68, 116.78, 103.94]" \
--scale_values "[58.40, 57.12, 57.38]"
* If the backbone of the model is Darknet53-FPN, use the following MO command line:
```sh
mo \
--input_model /path/to/yolact.onnx \
--reverse_input_channels \
--scale 255
```
.. code-block:: sh
mo \
--input_model /path/to/yolact.onnx \
--reverse_input_channels \
--scale 255
@endsphinxdirective