IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

DOCS shift to rst - Automatic Speech Recognition C++ Sample & Automatic Speech Recognition Python* Sample (#16609)

Browse Source
This commit is contained in:
Maciej Smyk
2023-03-29 16:39:09 +02:00
committed by GitHub
parent 10668f4f3a
commit 7d8f4af78a
2 changed files with 508 additions and 451 deletions
Download Patch File Download Diff File Expand all files Collapse all files
samples/cpp/speech_sample/README.md
+202 -161
View File
@@ -1,134 +1,164 @@
# Automatic Speech Recognition C++ Sample {#openvino_inference_engine_samples_speech_sample_README} # Automatic Speech Recognition C++ Sample {#openvino_inference_engine_samples_speech_sample_README}
@sphinxdirective
This sample demonstrates how to execute an Asynchronous Inference of acoustic model based on Kaldi\* neural networks and speech feature vectors. This sample demonstrates how to execute an Asynchronous Inference of acoustic model based on Kaldi\* neural networks and speech feature vectors.
The sample works with Kaldi ARK or Numpy* uncompressed NPZ files, so it does not cover an end-to-end speech recognition scenario (speech to text), requiring additional preprocessing (feature extraction) to get a feature vector from a speech signal, as well as postprocessing (decoding) to produce text from scores. The sample works with Kaldi ARK or Numpy* uncompressed NPZ files, so it does not cover an end-to-end speech recognition scenario (speech to text), requiring additional preprocessing (feature extraction) to get a feature vector from a speech signal, as well as postprocessing (decoding) to produce text from scores.
The following C++ API is used in the application: The following C++ API is used in the application:
| Feature | API | Description | +-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| :--- | :--- | :--- | | Feature | API | Description |
| Available Devices | `ov::Core::get_available_devices`, `ov::Core::get_property` | Get information of the devices for inference | +=============================================================+=============================================================================================================+==============================================================================+
| Import/Export Model | `ov::Core::import_model`, `ov::CompiledModel::export_model` | The GNA plugin supports loading and saving of the GNA-optimized model | | Available Devices | ``ov::Core::get_available_devices``, ``ov::Core::get_property`` | Get information of the devices for inference |
| Model Operations | `ov::set_batch`, `ov::Model::add_output`, `ov::CompiledModel::inputs`, `ov::CompiledModel::outputs` | Managing of model: configure batch_size, input and output tensors | +-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| Node Operations | `ov::OutputVector::size`, `ov::Output::get_shape` | Get node shape | | Import/Export Model | ``ov::Core::import_model``, ``ov::CompiledModel::export_model`` | The GNA plugin supports loading and saving of the GNA-optimized model |
| Asynchronous Infer | `ov::InferRequest::start_async`, `ov::InferRequest::wait` | Do asynchronous inference and waits until inference result becomes available | +-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| InferRequest Operations | `ov::InferRequest::query_state`, `ov::VariableState::reset` | Gets and resets CompiledModel state control | | Model Operations | ``ov::set_batch``, ``ov::Model::add_output``, ``ov::CompiledModel::inputs``, ``ov::CompiledModel::outputs`` | Managing of model: configure batch_size, input and output tensors |
| Tensor Operations | `ov::Tensor::get_size`, `ov::Tensor::data`, `ov::InferRequest::get_tensor` | Get a tensor, its size and data | +-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| Profiling | `ov::InferRequest::get_profiling_info` | Get infer request profiling info | | Node Operations | ``ov::OutputVector::size``, ``ov::Output::get_shape`` | Get node shape |
+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| Asynchronous Infer | ``ov::InferRequest::start_async``, ``ov::InferRequest::wait`` | Do asynchronous inference and waits until inference result becomes available |
+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| InferRequest Operations | ``ov::InferRequest::query_state``, ``ov::VariableState::reset`` | Gets and resets CompiledModel state control |
+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| Tensor Operations | ``ov::Tensor::get_size``, ``ov::Tensor::data``, ``ov::InferRequest::get_tensor`` | Get a tensor, its size and data |
+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
| Profiling | ``ov::InferRequest::get_profiling_info`` | Get infer request profiling info |
+-------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------+
Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md).
| Options | Values | Basic OpenVINO™ Runtime API is covered by :doc:`Hello Classification C++ sample <openvino_inference_engine_samples_hello_classification_README>`.
| :--- | :--- |
| Validated Models | Acoustic model based on Kaldi\* neural networks (see [Model Preparation](#model-preparation-speech) section) |
| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin) |
| Supported devices | See [Execution Modes](#execution-modes-speech) section below and [List Supported Devices](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) |
## How It Works +-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Options | Values |
+=============================================================+===============================================================================================================================================================+
| Validated Models | Acoustic model based on Kaldi\* neural networks (see :ref:`Model Preparation <model-preparation-speech>` section) |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin) |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Supported devices | See :ref:`Execution Modes <execution-modes-speech>` section below and :doc:`List Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
How It Works
############
At startup, the sample application reads command-line parameters, loads a specified model and input data to the OpenVINO™ Runtime plugin, performs inference on all speech utterances stored in the input file(s), logging each step in a standard output stream. At startup, the sample application reads command-line parameters, loads a specified model and input data to the OpenVINO™ Runtime plugin, performs inference on all speech utterances stored in the input file(s), logging each step in a standard output stream.
If the `-r` option is given, error statistics are provided for each speech utterance as shown above. If the ``-r`` option is given, error statistics are provided for each speech utterance as shown above.
You can see the explicit description of You can see the explicit description of
each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide. each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
### GNA-specific details GNA-specific details
++++++++++++++++++++
#### Quantization Quantization
------------
If the GNA device is selected (for example, using the `-d` GNA flag), the GNA OpenVINO™ Runtime plugin quantizes the model and input feature vector sequence to integer representation before performing inference. If the GNA device is selected (for example, using the ``-d`` GNA flag), the GNA OpenVINO™ Runtime plugin quantizes the model and input feature vector sequence to integer representation before performing inference.
Several parameters control neural network quantization. The `-q` flag determines the quantization mode. Several parameters control neural network quantization. The ``-q`` flag determines the quantization mode.
Two modes are supported: Two modes are supported:
- *static* - The first utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used for all subsequent inputs. The neural network is quantized to accommodate the scaled input dynamic range. - *static* - The first utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used for all subsequent inputs. The neural network is quantized to accommodate the scaled input dynamic range.
- *user-defined* - The user may specify a scale factor via the `-sf` flag that will be used for static quantization. - *user-defined* - The user may specify a scale factor via the ``-sf`` flag that will be used for static quantization.
The `-qb` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers. For example, when `-qb 8` is specified, the plugin will use 8-bit weights wherever possible in the The ``-qb`` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers. For example, when ``-qb 8`` is specified, the plugin will use 8-bit weights wherever possible in the
network. network.
> **NOTE**: .. note::
>
> - It is not always possible to use 8-bit weights due to GNA hardware limitations. For example, convolutional layers always use 16-bit weights (GNA hardware version 1 and 2). This limitation will be removed in GNA hardware version 3 and higher.
#### <a name="execution-modes-speech"></a> Execution Modes It is not always possible to use 8-bit weights due to GNA hardware limitations. For example, convolutional layers always use 16-bit weights (GNA hardware version 1 and 2). This limitation will be removed in GNA hardware version 3 and higher.
Several execution modes are supported via the `-d` flag:
- `CPU` - All calculations are performed on CPU device using CPU Plugin. .. _execution-modes-speech:
- `GPU` - All calculations are performed on GPU device using GPU Plugin.
- `VPUX` - All calculations are performed on VPUX device using VPUX Plugin.
- `GNA_AUTO` - GNA hardware is used if available and the driver is installed. Otherwise, the GNA device is emulated in fast-but-not-bit-exact mode.
- `GNA_HW` - GNA hardware is used if available and the driver is installed. Otherwise, an error will occur.
- `GNA_SW` - Deprecated. The GNA device is emulated in fast-but-not-bit-exact mode.
- `GNA_SW_FP32` - Substitutes parameters and calculations from low precision to floating point (FP32).
- `GNA_SW_EXACT` - GNA device is emulated in bit-exact mode.
#### Loading and Saving Models Execution Modes
---------------
The GNA plugin supports loading and saving of the GNA-optimized model (non-IR) via the `-rg` and `-wg` flags. Thereby, it is possible to avoid the cost of full model quantization at run time. The GNA plugin also supports export of firmware-compatible embedded model images for the Intel® Speech Enabling Developer Kit and Amazon Alexa* Premium Far-Field Voice Development Kit via the `-we` flag (save only). Several execution modes are supported via the ``-d`` flag:
- ``CPU`` - All calculations are performed on CPU device using CPU Plugin.
- ``GPU`` - All calculations are performed on GPU device using GPU Plugin.
- ``VPUX`` - All calculations are performed on VPUX device using VPUX Plugin.
- ``GNA_AUTO`` - GNA hardware is used if available and the driver is installed. Otherwise, the GNA device is emulated in fast-but-not-bit-exact mode.
- ``GNA_HW`` - GNA hardware is used if available and the driver is installed. Otherwise, an error will occur.
- ``GNA_SW`` - Deprecated. The GNA device is emulated in fast-but-not-bit-exact mode.
- ``GNA_SW_FP32`` - Substitutes parameters and calculations from low precision to floating point (FP32).
- ``GNA_SW_EXACT`` - GNA device is emulated in bit-exact mode.
Loading and Saving Models
-------------------------
The GNA plugin supports loading and saving of the GNA-optimized model (non-IR) via the ``-rg`` and ``-wg`` flags. Thereby, it is possible to avoid the cost of full model quantization at run time. The GNA plugin also supports export of firmware-compatible embedded model images for the Intel® Speech Enabling Developer Kit and Amazon Alexa* Premium Far-Field Voice Development Kit via the ``-we`` flag (save only).
In addition to performing inference directly from a GNA model file, these combinations of options make it possible to: In addition to performing inference directly from a GNA model file, these combinations of options make it possible to:
- Convert from IR format to GNA format model file (`-m`, `-wg`) - Convert from IR format to GNA format model file (``-m``, ``-wg``)
- Convert from IR format to embedded format model file (`-m`, `-we`) - Convert from IR format to embedded format model file (``-m``, ``-we``)
- Convert from GNA format to embedded format model file (`-rg`, `-we`) - Convert from GNA format to embedded format model file (``-rg``, ``-we``)
## Building Building
########
To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. To build the sample, please use instructions available at :doc:`Build the Sample Applications <openvino_docs_OV_UG_Samples_Overview>` section in OpenVINO™ Toolkit Samples guide.
## Running Running
#######
Run the application with the -h option to see the usage message: Run the application with the -h option to see the usage message:
``` .. code-block:: sh
speech_sample -h
``` speech_sample -h
Usage message: Usage message:
``` .. code-block:: sh
[ INFO ] OpenVINO Runtime version ......... <version>
[ INFO ] Build ........... <build> [ INFO ] OpenVINO Runtime version ......... <version>
[ INFO ] [ INFO ] Build ........... <build>
[ INFO ] Parsing input parameters [ INFO ]
[ INFO ] Parsing input parameters
speech_sample [OPTION]
Options:
-h Print a usage message.
-i "<path>" Required. Path(s) to input file(s). Usage for a single file/layer: <input_file.ark> or <input_file.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<input_file1.ark>,<layer2>:<port_num2>=<input_file2.ark>.
-m "<path>" Required. Path to an .xml file with a trained model (required if -rg is missing).
-o "<path>" Optional. Output file name(s) to save scores (inference results). Example of usage for a single file/layer: <output_file.ark> or <output_file.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<output_file1.ark>,<layer2>:<port_num2>=<output_file2.ark>.
-d "<device>" Optional. Specify a target device to infer on. CPU, GPU, VPUX, GNA_AUTO, GNA_HW, GNA_HW_WITH_SW_FBACK, GNA_SW_FP32, GNA_SW_EXACT and HETERO with combination of GNA as the primary device and CPU as a secondary (e.g. HETERO:GNA,CPU) are supported. The sample will look for a suitable plugin for device specified.
-pc Optional. Enables per-layer performance report.
-q "<mode>" Optional. Input quantization mode for GNA: static (default) or user defined (use with -sf).
-qb "<integer>" Optional. Weight resolution in bits for GNA quantization: 8 or 16 (default)
-sf "<double>" Optional. User-specified input scale factor for GNA quantization (use with -q user). If the model contains multiple inputs, provide scale factors by separating them with commas. For example: <layer1>:<sf1>,<layer2>:<sf2> or just <sf> to be applied to all inputs.
-bs "<integer>" Optional. Batch size 1-8 (default 1)
-r "<path>" Optional. Read reference score file(s) and compare inference results with reference scores. Usage for a single file/layer: <reference.ark> or <reference.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<reference_file1.ark>,<layer2>:<port_num2>=<reference_file2.ark>.
-rg "<path>" Read GNA model from file using path/filename provided (required if -m is missing).
-wg "<path>" Optional. Write GNA model to file using path/filename provided.
-we "<path>" Optional. Write GNA embedded model to file using path/filename provided.
-cw_l "<integer>" Optional. Number of frames for left context windows (default is 0). Works only with context window networks. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-cw_r "<integer>" Optional. Number of frames for right context windows (default is 0). Works only with context window networks. If you use the cw_r or cw_l flag, then batch size argument is ignored.
-layout "<string>" Optional. Prompts how network layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.
-pwl_me "<double>" Optional. The maximum percent of error for PWL function.The value must be in <0, 100> range. The default value is 1.0.
-exec_target "<string>" Optional. Specify GNA execution target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_EXEC_TARGET config option description.
-compile_target "<string>" Optional. Specify GNA compile target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_COMPILE_TARGET config option description.
-memory_reuse_off Optional. Disables memory optimizations for compiled model.
Available target devices: CPU GNA GPU VPUX
speech_sample [OPTION] .. _model-preparation-speech:
Options:
-h Print a usage message. Model Preparation
-i "<path>" Required. Path(s) to input file(s). Usage for a single file/layer: <input_file.ark> or <input_file.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<input_file1.ark>,<layer2>:<port_num2>=<input_file2.ark>. +++++++++++++++++
-m "<path>" Required. Path to an .xml file with a trained model (required if -rg is missing).
-o "<path>" Optional. Output file name(s) to save scores (inference results). Example of usage for a single file/layer: <output_file.ark> or <output_file.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<output_file1.ark>,<layer2>:<port_num2>=<output_file2.ark>.
-d "<device>" Optional. Specify a target device to infer on. CPU, GPU, VPUX, GNA_AUTO, GNA_HW, GNA_HW_WITH_SW_FBACK, GNA_SW_FP32, GNA_SW_EXACT and HETERO with combination of GNA as the primary device and CPU as a secondary (e.g. HETERO:GNA,CPU) are supported. The sample will look for a suitable plugin for device specified.
-pc Optional. Enables per-layer performance report.
-q "<mode>" Optional. Input quantization mode for GNA: static (default) or user defined (use with -sf).
-qb "<integer>" Optional. Weight resolution in bits for GNA quantization: 8 or 16 (default)
-sf "<double>" Optional. User-specified input scale factor for GNA quantization (use with -q user). If the model contains multiple inputs, provide scale factors by separating them with commas. For example: <layer1>:<sf1>,<layer2>:<sf2> or just <sf> to be applied to all inputs.
-bs "<integer>" Optional. Batch size 1-8 (default 1)
-r "<path>" Optional. Read reference score file(s) and compare inference results with reference scores. Usage for a single file/layer: <reference.ark> or <reference.npz>. Example of usage for several files/layers: <layer1>:<port_num1>=<reference_file1.ark>,<layer2>:<port_num2>=<reference_file2.ark>.
-rg "<path>" Read GNA model from file using path/filename provided (required if -m is missing).
-wg "<path>" Optional. Write GNA model to file using path/filename provided.
-we "<path>" Optional. Write GNA embedded model to file using path/filename provided.
-cw_l "<integer>" Optional. Number of frames for left context windows (default is 0). Works only with context window networks. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-cw_r "<integer>" Optional. Number of frames for right context windows (default is 0). Works only with context window networks. If you use the cw_r or cw_l flag, then batch size argument is ignored.
-layout "<string>" Optional. Prompts how network layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.
-pwl_me "<double>" Optional. The maximum percent of error for PWL function.The value must be in <0, 100> range. The default value is 1.0.
-exec_target "<string>" Optional. Specify GNA execution target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_EXEC_TARGET config option description.
-compile_target "<string>" Optional. Specify GNA compile target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_COMPILE_TARGET config option description.
-memory_reuse_off Optional. Disables memory optimizations for compiled model.
Available target devices: CPU GNA GPU VPUX
```
### <a name="model-preparation-speech"></a> Model Preparation
You can use the following model optimizer command to convert a Kaldi nnet1 or nnet2 neural model to OpenVINO™ toolkit Intermediate Representation format: You can use the following model optimizer command to convert a Kaldi nnet1 or nnet2 neural model to OpenVINO™ toolkit Intermediate Representation format:
``` .. code-block:: sh
mo --framework kaldi --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax --output_dir <OUTPUT_MODEL_DIR>
``` mo --framework kaldi --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax --output_dir <OUTPUT_MODEL_DIR>
The following pre-trained models are available: The following pre-trained models are available:
@@ -136,103 +166,114 @@ The following pre-trained models are available:
- rm_lstm4f - rm_lstm4f
- wsj_dnn5b_smbr - wsj_dnn5b_smbr
All of them can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2). All of them can be downloaded from `the storage <https://storage.openvinotoolkit.org/models_contrib/speech/2021.2>`__.
### Speech Inference Speech Inference
++++++++++++++++
Once the IR is created, you can do inference on Intel® Processors with the GNA co-processor (or emulation library): Once the IR is created, you can do inference on Intel® Processors with the GNA co-processor (or emulation library):
``` .. code-block:: sh
speech_sample -m wsj_dnn5b.xml -i dev93_10.ark -r dev93_scores_10.ark -d GNA_AUTO -o result.ark
``` speech_sample -m wsj_dnn5b.xml -i dev93_10.ark -r dev93_scores_10.ark -d GNA_AUTO -o result.ark
Here, the floating point Kaldi-generated reference neural network scores (`dev93_scores_10.ark`) corresponding to the input feature file (`dev93_10.ark`) are assumed to be available for comparison. Here, the floating point Kaldi-generated reference neural network scores (``dev93_scores_10.ark``) corresponding to the input feature file (``dev93_10.ark``) are assumed to be available for comparison.
> **NOTES**: .. note::
>
> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
>
> - The sample supports input and output in numpy file format (.npz)
## Sample Output - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the :doc:`Model Optimizer tool <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
- The sample supports input and output in numpy file format (.npz)
Sample Output
#############
The sample application logs each step in a standard output stream. The sample application logs each step in a standard output stream.
``` .. code-block:: sh
[ INFO ] OpenVINO runtime: OpenVINO Runtime version ......... 2022.1.0
[ INFO ] Build ........... 2022.1.0-6311-a90bb1ff017 [ INFO ] OpenVINO runtime: OpenVINO Runtime version ......... 2022.1.0
[ INFO ] [ INFO ] Build ........... 2022.1.0-6311-a90bb1ff017
[ INFO ] Parsing input parameters [ INFO ]
[ INFO ] Loading model files: [ INFO ] Parsing input parameters
[ INFO ] \test_data\models\wsj_dnn5b_smbr_fp32\wsj_dnn5b_smbr_fp32.xml [ INFO ] Loading model files:
[ INFO ] Using scale factor of 2175.43 calculated from first utterance. [ INFO ] \test_data\models\wsj_dnn5b_smbr_fp32\wsj_dnn5b_smbr_fp32.xml
[ INFO ] Model loading time 0.0034 ms [ INFO ] Using scale factor of 2175.43 calculated from first utterance.
[ INFO ] Loading model to the device GNA_AUTO [ INFO ] Model loading time 0.0034 ms
[ INFO ] Loading model to the device [ INFO ] Loading model to the device GNA_AUTO
[ INFO ] Number scores per frame : 3425 [ INFO ] Loading model to the device
Utterance 0: [ INFO ] Number scores per frame : 3425
Total time in Infer (HW and SW): 5687.53 ms Utterance 0:
Frames in utterance: 1294 frames Total time in Infer (HW and SW): 5687.53 ms
Average Infer time per frame: 4.39531 ms Frames in utterance: 1294 frames
max error: 0.705184 Average Infer time per frame: 4.39531 ms
avg error: 0.0448388 max error: 0.705184
avg rms error: 0.0574098 avg error: 0.0448388
stdev error: 0.0371649 avg rms error: 0.0574098
stdev error: 0.0371649
End of Utterance 0
[ INFO ] Number scores per frame : 3425
Utterance 1:
Total time in Infer (HW and SW): 4341.34 ms
Frames in utterance: 1005 frames
Average Infer time per frame: 4.31974 ms
max error: 0.757597
avg error: 0.0452166
avg rms error: 0.0578436
stdev error: 0.0372769
End of Utterance 1
...
End of Utterance X
[ INFO ] Execution successful
Use of Sample in Kaldi* Speech Recognition Pipeline
###################################################
End of Utterance 0 The Wall Street Journal DNN model used in this example was prepared using the Kaldi s5 recipe and the Kaldi Nnet (nnet1) framework. It is possible to recognize speech by substituting the ``speech_sample`` for
Kaldi's nnet-forward command. Since the ``speech_sample`` does not yet use pipes, it is necessary to use temporary files for speaker-transformed feature vectors and scores when running the Kaldi speech recognition pipeline. The following operations assume that feature extraction was already performed according to the ``s5`` recipe and that the working directory within the Kaldi source tree is ``egs/wsj/s5``.
[ INFO ] Number scores per frame : 3425 1. Prepare a speaker-transformed feature set given the feature transform specified in ``final.feature_transform`` and the feature files specified in ``feats.scp``:
Utterance 1:
Total time in Infer (HW and SW): 4341.34 ms .. code-block:: sh
Frames in utterance: 1005 frames
Average Infer time per frame: 4.31974 ms nnet-forward --use-gpu=no final.feature_transform "ark,s,cs:copy-feats scp:feats.scp ark:- |" ark:feat.ark
max error: 0.757597
avg error: 0.0452166
avg rms error: 0.0578436
stdev error: 0.0372769
2. Score the feature set using the ``speech_sample``:
End of Utterance 1 .. code-block:: sh
./speech_sample -d GNA_AUTO -bs 8 -i feat.ark -m wsj_dnn5b.xml -o scores.ark
... OpenVINO™ toolkit Intermediate Representation ``wsj_dnn5b.xml`` file was generated in the previous :ref:`Model Preparation <model-preparation-speech>` section.
End of Utterance X
[ INFO ] Execution successful 3. Run the Kaldi decoder to produce n-best text hypotheses and select most likely text given the WFST (``HCLG.fst``), vocabulary (``words.txt``), and TID/PID mapping (``final.mdl``):
```
.. code-block:: sh
latgen-faster-mapped --max-active=7000 --max-mem=50000000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.0833 --allow-partial=true --word-symbol-table=words.txt final.mdl HCLG.fst ark:scores.ark ark:-| lattice-scale --inv-acoustic-scale=13 ark:- ark:- | lattice-best-path --word-symbol-table=words.txt ark:- ark,t:- > out.txt &
## Use of Sample in Kaldi* Speech Recognition Pipeline 4. Run the word error rate tool to check accuracy given the vocabulary (``words.txt``) and reference transcript (``test_filt.txt``):
The Wall Street Journal DNN model used in this example was prepared using the Kaldi s5 recipe and the Kaldi Nnet (nnet1) framework. It is possible to recognize speech by substituting the `speech_sample` for .. code-block:: sh
Kaldi's nnet-forward command. Since the `speech_sample` does not yet use pipes, it is necessary to use temporary files for speaker-transformed feature vectors and scores when running the Kaldi speech recognition pipeline. The following operations assume that feature extraction was already performed according to the `s5` recipe and that the working directory within the Kaldi source tree is `egs/wsj/s5`.
cat out.txt | utils/int2sym.pl -f 2- words.txt | sed s:\<UNK\>::g | compute-wer --text --mode=present ark:test_filt.txt ark,p:-
1. Prepare a speaker-transformed feature set given the feature transform specified in `final.feature_transform` and the feature files specified in `feats.scp`: All of mentioned files can be downloaded from `the storage <https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/wsj_dnn5b_smbr>`__
```sh
nnet-forward --use-gpu=no final.feature_transform "ark,s,cs:copy-feats scp:feats.scp ark:- |" ark:feat.ark
```
2. Score the feature set using the `speech_sample`: See Also
```sh ########
./speech_sample -d GNA_AUTO -bs 8 -i feat.ark -m wsj_dnn5b.xml -o scores.ark
```
OpenVINO™ toolkit Intermediate Representation `wsj_dnn5b.xml` file was generated in the previous [Model Preparation](#model-preparation-speech) section. - :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
- :doc:`Using OpenVINO™ Toolkit Samples <openvino_docs_OV_UG_Samples_Overview>`
- :doc:`Model Downloader <omz_tools_downloader>`
- :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
3. Run the Kaldi decoder to produce n-best text hypotheses and select most likely text given the WFST (`HCLG.fst`), vocabulary (`words.txt`), and TID/PID mapping (`final.mdl`): @endsphinxdirective
```sh
latgen-faster-mapped --max-active=7000 --max-mem=50000000 --beam=13.0 --lattice-beam=6.0 --acoustic-scale=0.0833 --allow-partial=true --word-symbol-table=words.txt final.mdl HCLG.fst ark:scores.ark ark:-| lattice-scale --inv-acoustic-scale=13 ark:- ark:- | lattice-best-path --word-symbol-table=words.txt ark:- ark,t:- > out.txt &
```
4. Run the word error rate tool to check accuracy given the vocabulary (`words.txt`) and reference transcript (`test_filt.txt`):
```sh
cat out.txt | utils/int2sym.pl -f 2- words.txt | sed s:\<UNK\>::g | compute-wer --text --mode=present ark:test_filt.txt ark,p:-
```
All of mentioned files can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/wsj_dnn5b_smbr](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/wsj_dnn5b_smbr)
## See Also
- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/integrate_with_your_application.md)
- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md)
- [Model Downloader](@ref omz_tools_downloader)
- [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)
samples/python/speech_sample/README.md
+306 -290
View File
@@ -1,164 +1,190 @@
# Automatic Speech Recognition Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_speech_sample_README} # Automatic Speech Recognition Python* Sample {#openvino_inference_engine_ie_bridges_python_sample_speech_sample_README}
@sphinxdirective
This sample demonstrates how to do a Synchronous Inference of acoustic model based on Kaldi\* neural models and speech feature vectors. This sample demonstrates how to do a Synchronous Inference of acoustic model based on Kaldi\* neural models and speech feature vectors.
The sample works with Kaldi ARK or Numpy* uncompressed NPZ files, so it does not cover an end-to-end speech recognition scenario (speech to text), requiring additional preprocessing (feature extraction) to get a feature vector from a speech signal, as well as postprocessing (decoding) to produce text from scores. The sample works with Kaldi ARK or Numpy* uncompressed NPZ files, so it does not cover an end-to-end speech recognition scenario (speech to text), requiring additional preprocessing (feature extraction) to get a feature vector from a speech signal, as well as postprocessing (decoding) to produce text from scores.
Automatic Speech Recognition Python sample application demonstrates how to use the following Python API in applications: Automatic Speech Recognition Python sample application demonstrates how to use the following Python API in applications:
| Feature | API | Description | +-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------- | | Feature | API | Description |
| Import/Export Model | [openvino.runtime.Core.import_model], [openvino.runtime.CompiledModel.export_model] | The GNA plugin supports loading and saving of the GNA-optimized model | +===================================================================+================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+=======================================================================+
| Model Operations | [openvino.runtime.Model.add_outputs], [openvino.runtime.set_batch], [openvino.runtime.CompiledModel.inputs], [openvino.runtime.CompiledModel.outputs], [openvino.runtime.ConstOutput.any_name] | Managing of model: configure batch_size, input and output tensors | | Import/Export Model | `openvino.runtime.Core.import_model <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.Core.html#openvino.runtime.Core.import_model>`__ , `openvino.runtime.CompiledModel.export_model <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.export_model>`__ | The GNA plugin supports loading and saving of the GNA-optimized model |
| Synchronous Infer | [openvino.runtime.CompiledModel.create_infer_request], [openvino.runtime.InferRequest.infer] | Do synchronous inference | +-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| InferRequest Operations | [openvino.runtime.InferRequest.get_input_tensor], [openvino.runtime.InferRequest.model_outputs], [openvino.runtime.InferRequest.model_inputs], | Get info about model using infer request API | | Model Operations | `openvino.runtime.Model.add_outputs <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.add_outputs>`__ , `openvino.runtime.set_batch <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.html#openvino.runtime.set_batch>`__ , `openvino.runtime.CompiledModel.inputs <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.inputs>`__ , `openvino.runtime.CompiledModel.outputs <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.outputs>`__ , `openvino.runtime.ConstOutput.any_name <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.ConstOutput.html#openvino.runtime.ConstOutput.any_name>`__ | Managing of model: configure batch_size, input and output tensors |
| InferRequest Operations | [openvino.runtime.InferRequest.query_state], [openvino.runtime.VariableState.reset] | Gets and resets CompiledModel state control | +-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| Profiling | [openvino.runtime.InferRequest.profiling_info], [openvino.runtime.ProfilingInfo.real_time] | Get infer request profiling info | | Synchronous Infer | `openvino.runtime.CompiledModel.create_infer_request <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.create_infer_request>`__ , `openvino.runtime.InferRequest.infer <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.infer>`__ | Do synchronous inference |
+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| InferRequest Operations | `openvino.runtime.InferRequest.get_input_tensor <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.get_input_tensor>`__ , `openvino.runtime.InferRequest.model_outputs <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.model_outputs>`__ , `openvino.runtime.InferRequest.model_inputs <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.model_inputs>`__ , | Get info about model using infer request API |
+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| InferRequest Operations | `openvino.runtime.InferRequest.query_state <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.query_state>`__ , `openvino.runtime.VariableState.reset <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.inference_engine.VariableState.html#openvino.inference_engine.VariableState.reset>`__ | Gets and resets CompiledModel state control |
+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
| Profiling | `openvino.runtime.InferRequest.profiling_info <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.profiling_info>`__ , `openvino.runtime.ProfilingInfo.real_time <https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.ProfilingInfo.html#openvino.runtime.ProfilingInfo.real_time>`__ | Get infer request profiling info |
+-------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------+
Basic OpenVINO™ Runtime API is covered by [Hello Classification Python* Sample](../hello_classification/README.md). Basic OpenVINO™ Runtime API is covered by :doc:`Hello Classification Python* Sample <openvino_inference_engine_ie_bridges_python_sample_hello_classification_README>`.
| Options | Values | +----------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| :------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------- | | Options | Values |
| Validated Models | Acoustic model based on Kaldi* neural models (see [Model Preparation](#model-preparation-speech-python) section) | +======================================================================+======================================================================================================================================================================+
| Model Format | OpenVINO™ toolkit Intermediate Representation (.xml + .bin) | | Validated Models | Acoustic model based on Kaldi* neural models (see :ref:`Model Preparation <model-preparation-speech-python>` section) |
| Supported devices | See [Execution Modes](#execution-modes-speech-python) section below and [List Supported Devices](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +----------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Other language realization | [C++](../../../samples/cpp/speech_sample/README.md) | | Model Format | OpenVINO™ toolkit Intermediate Representation (.xml + .bin) |
+----------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Supported devices | See :ref:`Execution Modes <execution-modes-speech-python>` section below and :doc:`List Supported Devices <openvino_docs_OV_UG_supported_plugins_Supported_Devices>` |
+----------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Other language realization | :doc:`C++ <openvino_inference_engine_samples_speech_sample_README>` |
+----------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
## How It Works
How It Works
############
At startup, the sample application reads command-line parameters, loads a specified model and input data to the OpenVINO™ Runtime plugin, performs synchronous inference on all speech utterances stored in the input file, logging each step in a standard output stream. At startup, the sample application reads command-line parameters, loads a specified model and input data to the OpenVINO™ Runtime plugin, performs synchronous inference on all speech utterances stored in the input file, logging each step in a standard output stream.
You can see the explicit description of You can see the explicit description of
each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/integrate_with_your_application.md) section of "Integrate OpenVINO™ Runtime with Your Application" guide. each sample step at :doc:`Integration Steps <openvino_docs_OV_UG_Integrate_OV_with_your_application>` section of "Integrate OpenVINO™ Runtime with Your Application" guide.
## GNA-specific details GNA-specific details
####################
### Quantization Quantization
++++++++++++
If the GNA device is selected (for example, using the `-d` GNA flag), the GNA OpenVINO™ Runtime plugin quantizes the model and input feature vector sequence to integer representation before performing inference. If the GNA device is selected (for example, using the ``-d`` GNA flag), the GNA OpenVINO™ Runtime plugin quantizes the model and input feature vector sequence to integer representation before performing inference.
Several neural model quantization modes: Several neural model quantization modes:
- *static* - The first utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used for all subsequent inputs. The neural model is quantized to accommodate the scaled input dynamic range. - *static* - The first utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used for all subsequent inputs. The neural model is quantized to accommodate the scaled input dynamic range.
- *user-defined* - The user may specify a scale factor via the `-sf` flag that will be used for static quantization. - *user-defined* - The user may specify a scale factor via the ``-sf`` flag that will be used for static quantization.
The `-qb` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers. The ``-qb`` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers.
For example, when `-qb 8` is specified, the plugin will use 8-bit weights wherever possible in the For example, when ``-qb 8`` is specified, the plugin will use 8-bit weights wherever possible in the
model. model.
> **NOTE**: .. note::
>
> - It is not always possible to use 8-bit weights due to GNA hardware limitations. For example, convolutional layers always use 16-bit weights (GNA hardware version 1 and 2). This limitation will be removed in GNA hardware version 3 and higher.
>
### <a name="execution-modes-speech-python"></a> Execution Modes It is not always possible to use 8-bit weights due to GNA hardware limitations. For example, convolutional layers always use 16-bit weights (GNA hardware version 1 and 2). This limitation will be removed in GNA hardware version 3 and higher.
Several execution modes are supported via the `-d` flag: .. _execution-modes-speech-python:
- `CPU` - All calculations are performed on CPU device using CPU Plugin. Execution Modes
- `GPU` - All calculations are performed on GPU device using GPU Plugin. +++++++++++++++
- `VPUX` - All calculations are performed on VPUX device using VPUX Plugin.
- `GNA_AUTO` - GNA hardware is used if available and the driver is installed. Otherwise, the GNA device is emulated in fast-but-not-bit-exact mode.
- `GNA_HW` - GNA hardware is used if available and the driver is installed. Otherwise, an error will occur.
- `GNA_SW` - Deprecated. The GNA device is emulated in fast-but-not-bit-exact mode.
- `GNA_SW_FP32` - Substitutes parameters and calculations from low precision to floating point (FP32).
- `GNA_SW_EXACT` - GNA device is emulated in bit-exact mode.
### Loading and Saving Models Several execution modes are supported via the ``-d`` flag:
The GNA plugin supports loading and saving of the GNA-optimized model (non-IR) via the `-rg` and `-wg` flags. - ``CPU`` - All calculations are performed on CPU device using CPU Plugin.
- ``GPU`` - All calculations are performed on GPU device using GPU Plugin.
- ``VPUX`` - All calculations are performed on VPUX device using VPUX Plugin.
- ``GNA_AUTO`` - GNA hardware is used if available and the driver is installed. Otherwise, the GNA device is emulated in fast-but-not-bit-exact mode.
- ``GNA_HW`` - GNA hardware is used if available and the driver is installed. Otherwise, an error will occur.
- ``GNA_SW`` - Deprecated. The GNA device is emulated in fast-but-not-bit-exact mode.
- ``GNA_SW_FP32`` - Substitutes parameters and calculations from low precision to floating point (FP32).
- ``GNA_SW_EXACT`` - GNA device is emulated in bit-exact mode.
Loading and Saving Models
+++++++++++++++++++++++++
The GNA plugin supports loading and saving of the GNA-optimized model (non-IR) via the ``-rg`` and ``-wg`` flags.
Thereby, it is possible to avoid the cost of full model quantization at run time. Thereby, it is possible to avoid the cost of full model quantization at run time.
The GNA plugin also supports export of firmware-compatible embedded model images for the Intel® Speech Enabling Developer Kit and Amazon Alexa* Premium Far-Field Voice Development Kit via the `-we` flag (save only). The GNA plugin also supports export of firmware-compatible embedded model images for the Intel® Speech Enabling Developer Kit and Amazon Alexa* Premium Far-Field Voice Development Kit via the ``-we`` flag (save only).
In addition to performing inference directly from a GNA model file, these options make it possible to: In addition to performing inference directly from a GNA model file, these options make it possible to:
- Convert from IR format to GNA format model file (`-m`, `-wg`) - Convert from IR format to GNA format model file (``-m``, ``-wg``)
- Convert from IR format to embedded format model file (`-m`, `-we`) - Convert from IR format to embedded format model file (``-m``, ``-we``)
- Convert from GNA format to embedded format model file (`-rg`, `-we`) - Convert from GNA format to embedded format model file (``-rg``, ``-we``)
## Running Running
#######
Run the application with the `-h` option to see the usage message: Run the application with the ``-h`` option to see the usage message:
``` .. code-block:: bash
python speech_sample.py -h
``` python speech_sample.py -h
Usage message: Usage message:
``` .. code-block:: console
usage: speech_sample.py [-h] (-m MODEL | -rg IMPORT_GNA_MODEL) -i INPUT [-o OUTPUT] [-r REFERENCE] [-d DEVICE] [-bs [1-8]]
[-layout LAYOUT] [-qb [8, 16]] [-sf SCALE_FACTOR] [-wg EXPORT_GNA_MODEL] usage: speech_sample.py [-h] (-m MODEL | -rg IMPORT_GNA_MODEL) -i INPUT [-o OUTPUT] [-r REFERENCE] [-d DEVICE] [-bs [1-8]]
[-we EXPORT_EMBEDDED_GNA_MODEL] [-we_gen [GNA1, GNA3]] [-layout LAYOUT] [-qb [8, 16]] [-sf SCALE_FACTOR] [-wg EXPORT_GNA_MODEL]
[--exec_target [GNA_TARGET_2_0, GNA_TARGET_3_0]] [-pc] [-a [CORE, ATOM]] [-iname INPUT_LAYERS] [-we EXPORT_EMBEDDED_GNA_MODEL] [-we_gen [GNA1, GNA3]]
[-oname OUTPUT_LAYERS] [-cw_l CONTEXT_WINDOW_LEFT] [-cw_r CONTEXT_WINDOW_RIGHT] [-pwl_me PWL_ME] [--exec_target [GNA_TARGET_2_0, GNA_TARGET_3_0]] [-pc] [-a [CORE, ATOM]] [-iname INPUT_LAYERS]
[-oname OUTPUT_LAYERS] [-cw_l CONTEXT_WINDOW_LEFT] [-cw_r CONTEXT_WINDOW_RIGHT] [-pwl_me PWL_ME]
optional arguments:
-m MODEL, --model MODEL
Path to an .xml file with a trained model (required if -rg is missing).
-rg IMPORT_GNA_MODEL, --import_gna_model IMPORT_GNA_MODEL
Read GNA model from file using path/filename provided (required if -m is missing).
Options:
-h, --help Show this help message and exit.
-i INPUT, --input INPUT
Required. Path(s) to input file(s).
Usage for a single file/layer: <input_file.ark> or <input_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<input_file1.ark>,<layer2>:<port_num2>=<input_file2.ark>.
-o OUTPUT, --output OUTPUT
Optional. Output file name(s) to save scores (inference results).
Usage for a single file/layer: <output_file.ark> or <output_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<output_file1.ark>,<layer2>:<port_num2>=<output_file2.ark>.
-r REFERENCE, --reference REFERENCE
Read reference score file(s) and compare inference results with reference scores.
Usage for a single file/layer: <reference_file.ark> or <reference_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<reference_file1.ark>,<layer2>:<port_num2>=<reference_file2.ark>.
-d DEVICE, --device DEVICE
Optional. Specify a target device to infer on. CPU, GPU, VPUX, GNA_AUTO, GNA_HW, GNA_SW_FP32,
GNA_SW_EXACT and HETERO with combination of GNA as the primary device and CPU as a secondary (e.g.
HETERO:GNA,CPU) are supported. The sample will look for a suitable plugin for device specified.
Default value is CPU.
-bs [1-8], --batch_size [1-8]
Optional. Batch size 1-8.
-layout LAYOUT Optional. Custom layout in format: "input0[value0],input1[value1]" or "[value]" (applied to all
inputs)
-qb [8, 16], --quantization_bits [8, 16]
Optional. Weight resolution in bits for GNA quantization: 8 or 16 (default 16).
-sf SCALE_FACTOR, --scale_factor SCALE_FACTOR
Optional. User-specified input scale factor for GNA quantization.
If the model contains multiple inputs, provide scale factors by separating them with commas.
For example: <layer1>:<sf1>,<layer2>:<sf2> or just <sf> to be applied to all inputs.
-wg EXPORT_GNA_MODEL, --export_gna_model EXPORT_GNA_MODEL
Optional. Write GNA model to file using path/filename provided.
-we EXPORT_EMBEDDED_GNA_MODEL, --export_embedded_gna_model EXPORT_EMBEDDED_GNA_MODEL
Optional. Write GNA embedded model to file using path/filename provided.
-we_gen [GNA1, GNA3], --embedded_gna_configuration [GNA1, GNA3]
Optional. GNA generation configuration string for embedded export. Can be GNA1 (default) or GNA3.
--exec_target [GNA_TARGET_2_0, GNA_TARGET_3_0]
Optional. Specify GNA execution target generation. By default, generation corresponds to the GNA HW
available in the system or the latest fully supported generation by the software. See the GNA
Plugin's GNA_EXEC_TARGET config option description.
-pc, --performance_counter
Optional. Enables performance report (specify -a to ensure arch accurate results).
-a [CORE, ATOM], --arch [CORE, ATOM]
Optional. Specify architecture. CORE, ATOM with the combination of -pc.
-cw_l CONTEXT_WINDOW_LEFT, --context_window_left CONTEXT_WINDOW_LEFT
Optional. Number of frames for left context windows (default is 0). Works only with context window
models. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-cw_r CONTEXT_WINDOW_RIGHT, --context_window_right CONTEXT_WINDOW_RIGHT
Optional. Number of frames for right context windows (default is 0). Works only with context window
models. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-pwl_me PWL_ME Optional. The maximum percent of error for PWL function. The value must be in <0, 100> range. The
default value is 1.0.
optional arguments: .. _model-preparation-speech-python:
-m MODEL, --model MODEL
Path to an .xml file with a trained model (required if -rg is missing).
-rg IMPORT_GNA_MODEL, --import_gna_model IMPORT_GNA_MODEL
Read GNA model from file using path/filename provided (required if -m is missing).
Options: Model Preparation
-h, --help Show this help message and exit. #################
-i INPUT, --input INPUT
Required. Path(s) to input file(s).
Usage for a single file/layer: <input_file.ark> or <input_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<input_file1.ark>,<layer2>:<port_num2>=<input_file2.ark>.
-o OUTPUT, --output OUTPUT
Optional. Output file name(s) to save scores (inference results).
Usage for a single file/layer: <output_file.ark> or <output_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<output_file1.ark>,<layer2>:<port_num2>=<output_file2.ark>.
-r REFERENCE, --reference REFERENCE
Read reference score file(s) and compare inference results with reference scores.
Usage for a single file/layer: <reference_file.ark> or <reference_file.npz>.
Example of usage for several files/layers: <layer1>:<port_num1>=<reference_file1.ark>,<layer2>:<port_num2>=<reference_file2.ark>.
-d DEVICE, --device DEVICE
Optional. Specify a target device to infer on. CPU, GPU, VPUX, GNA_AUTO, GNA_HW, GNA_SW_FP32,
GNA_SW_EXACT and HETERO with combination of GNA as the primary device and CPU as a secondary (e.g.
HETERO:GNA,CPU) are supported. The sample will look for a suitable plugin for device specified.
Default value is CPU.
-bs [1-8], --batch_size [1-8]
Optional. Batch size 1-8.
-layout LAYOUT Optional. Custom layout in format: "input0[value0],input1[value1]" or "[value]" (applied to all
inputs)
-qb [8, 16], --quantization_bits [8, 16]
Optional. Weight resolution in bits for GNA quantization: 8 or 16 (default 16).
-sf SCALE_FACTOR, --scale_factor SCALE_FACTOR
Optional. User-specified input scale factor for GNA quantization.
If the model contains multiple inputs, provide scale factors by separating them with commas.
For example: <layer1>:<sf1>,<layer2>:<sf2> or just <sf> to be applied to all inputs.
-wg EXPORT_GNA_MODEL, --export_gna_model EXPORT_GNA_MODEL
Optional. Write GNA model to file using path/filename provided.
-we EXPORT_EMBEDDED_GNA_MODEL, --export_embedded_gna_model EXPORT_EMBEDDED_GNA_MODEL
Optional. Write GNA embedded model to file using path/filename provided.
-we_gen [GNA1, GNA3], --embedded_gna_configuration [GNA1, GNA3]
Optional. GNA generation configuration string for embedded export. Can be GNA1 (default) or GNA3.
--exec_target [GNA_TARGET_2_0, GNA_TARGET_3_0]
Optional. Specify GNA execution target generation. By default, generation corresponds to the GNA HW
available in the system or the latest fully supported generation by the software. See the GNA
Plugin's GNA_EXEC_TARGET config option description.
-pc, --performance_counter
Optional. Enables performance report (specify -a to ensure arch accurate results).
-a [CORE, ATOM], --arch [CORE, ATOM]
Optional. Specify architecture. CORE, ATOM with the combination of -pc.
-cw_l CONTEXT_WINDOW_LEFT, --context_window_left CONTEXT_WINDOW_LEFT
Optional. Number of frames for left context windows (default is 0). Works only with context window
models. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-cw_r CONTEXT_WINDOW_RIGHT, --context_window_right CONTEXT_WINDOW_RIGHT
Optional. Number of frames for right context windows (default is 0). Works only with context window
models. If you use the cw_l or cw_r flag, then batch size argument is ignored.
-pwl_me PWL_ME Optional. The maximum percent of error for PWL function. The value must be in <0, 100> range. The
default value is 1.0.
```
## <a name="model-preparation-speech-python"></a> Model Preparation
You can use the following model optimizer command to convert a Kaldi nnet1 or nnet2 neural model to OpenVINO™ toolkit Intermediate Representation format: You can use the following model optimizer command to convert a Kaldi nnet1 or nnet2 neural model to OpenVINO™ toolkit Intermediate Representation format:
```sh .. code-block:: sh
mo --framework kaldi --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax --output_dir <OUTPUT_MODEL_DIR>
``` mo --framework kaldi --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax --output_dir <OUTPUT_MODEL_DIR>
The following pre-trained models are available: The following pre-trained models are available:
@@ -166,189 +192,179 @@ The following pre-trained models are available:
- rm_lstm4f - rm_lstm4f
- wsj_dnn5b_smbr - wsj_dnn5b_smbr
All of them can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2). All of them can be downloaded from `the storage <https://storage.openvinotoolkit.org/models_contrib/speech/2021.2>`.
## Speech Inference Speech Inference
################
You can do inference on Intel® Processors with the GNA co-processor (or emulation library): You can do inference on Intel® Processors with the GNA co-processor (or emulation library):
``` .. code-block:: sh
python speech_sample.py -m wsj_dnn5b.xml -i dev93_10.ark -r dev93_scores_10.ark -d GNA_AUTO -o result.npz
``` python speech_sample.py -m wsj_dnn5b.xml -i dev93_10.ark -r dev93_scores_10.ark -d GNA_AUTO -o result.npz
> **NOTES**:
>
> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md).
>
> - The sample supports input and output in numpy file format (.npz)
## Sample Output .. note::
- Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the :doc:`Model Optimizer tool <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`.
- The sample supports input and output in numpy file format (.npz)
Sample Output
#############
The sample application logs each step in a standard output stream. The sample application logs each step in a standard output stream.
``` .. code-block:: sh
[ INFO ] Creating OpenVINO Runtime Core
[ INFO ] Reading the model: /models/wsj_dnn5b_smbr_fp32.xml [ INFO ] Creating OpenVINO Runtime Core
[ INFO ] Using scale factor(s) calculated from first utterance [ INFO ] Reading the model: /models/wsj_dnn5b_smbr_fp32.xml
[ INFO ] For input 0 using scale factor of 2175.4322418 [ INFO ] Using scale factor(s) calculated from first utterance
[ INFO ] Loading the model to the plugin [ INFO ] For input 0 using scale factor of 2175.4322418
[ INFO ] Starting inference in synchronous mode [ INFO ] Loading the model to the plugin
[ INFO ] [ INFO ] Starting inference in synchronous mode
[ INFO ] Utterance 0: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 6326.06ms [ INFO ] Utterance 0:
[ INFO ] Frames in utterance: 1294 [ INFO ] Total time in Infer (HW and SW): 6326.06ms
[ INFO ] Average Infer time per frame: 4.89ms [ INFO ] Frames in utterance: 1294
[ INFO ] [ INFO ] Average Infer time per frame: 4.89ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7051840 [ INFO ]
[ INFO ] avg error: 0.0448388 [ INFO ] max error: 0.7051840
[ INFO ] avg rms error: 0.0582387 [ INFO ] avg error: 0.0448388
[ INFO ] stdev error: 0.0371650 [ INFO ] avg rms error: 0.0582387
[ INFO ] [ INFO ] stdev error: 0.0371650
[ INFO ] Utterance 1: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 4526.57ms [ INFO ] Utterance 1:
[ INFO ] Frames in utterance: 1005 [ INFO ] Total time in Infer (HW and SW): 4526.57ms
[ INFO ] Average Infer time per frame: 4.50ms [ INFO ] Frames in utterance: 1005
[ INFO ] [ INFO ] Average Infer time per frame: 4.50ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7575974 [ INFO ]
[ INFO ] avg error: 0.0452166 [ INFO ] max error: 0.7575974
[ INFO ] avg rms error: 0.0586013 [ INFO ] avg error: 0.0452166
[ INFO ] stdev error: 0.0372769 [ INFO ] avg rms error: 0.0586013
[ INFO ] [ INFO ] stdev error: 0.0372769
[ INFO ] Utterance 2: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 6636.56ms [ INFO ] Utterance 2:
[ INFO ] Frames in utterance: 1471 [ INFO ] Total time in Infer (HW and SW): 6636.56ms
[ INFO ] Average Infer time per frame: 4.51ms [ INFO ] Frames in utterance: 1471
[ INFO ] [ INFO ] Average Infer time per frame: 4.51ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7191710 [ INFO ]
[ INFO ] avg error: 0.0472226 [ INFO ] max error: 0.7191710
[ INFO ] avg rms error: 0.0612991 [ INFO ] avg error: 0.0472226
[ INFO ] stdev error: 0.0390846 [ INFO ] avg rms error: 0.0612991
[ INFO ] [ INFO ] stdev error: 0.0390846
[ INFO ] Utterance 3: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 3927.01ms [ INFO ] Utterance 3:
[ INFO ] Frames in utterance: 845 [ INFO ] Total time in Infer (HW and SW): 3927.01ms
[ INFO ] Average Infer time per frame: 4.65ms [ INFO ] Frames in utterance: 845
[ INFO ] [ INFO ] Average Infer time per frame: 4.65ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7436461 [ INFO ]
[ INFO ] avg error: 0.0477581 [ INFO ] max error: 0.7436461
[ INFO ] avg rms error: 0.0621334 [ INFO ] avg error: 0.0477581
[ INFO ] stdev error: 0.0397457 [ INFO ] avg rms error: 0.0621334
[ INFO ] [ INFO ] stdev error: 0.0397457
[ INFO ] Utterance 4: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 3891.49ms [ INFO ] Utterance 4:
[ INFO ] Frames in utterance: 855 [ INFO ] Total time in Infer (HW and SW): 3891.49ms
[ INFO ] Average Infer time per frame: 4.55ms [ INFO ] Frames in utterance: 855
[ INFO ] [ INFO ] Average Infer time per frame: 4.55ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7071600 [ INFO ]
[ INFO ] avg error: 0.0449147 [ INFO ] max error: 0.7071600
[ INFO ] avg rms error: 0.0585048 [ INFO ] avg error: 0.0449147
[ INFO ] stdev error: 0.0374897 [ INFO ] avg rms error: 0.0585048
[ INFO ] [ INFO ] stdev error: 0.0374897
[ INFO ] Utterance 5: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 3378.61ms [ INFO ] Utterance 5:
[ INFO ] Frames in utterance: 699 [ INFO ] Total time in Infer (HW and SW): 3378.61ms
[ INFO ] Average Infer time per frame: 4.83ms [ INFO ] Frames in utterance: 699
[ INFO ] [ INFO ] Average Infer time per frame: 4.83ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.8870468 [ INFO ]
[ INFO ] avg error: 0.0479243 [ INFO ] max error: 0.8870468
[ INFO ] avg rms error: 0.0625490 [ INFO ] avg error: 0.0479243
[ INFO ] stdev error: 0.0401951 [ INFO ] avg rms error: 0.0625490
[ INFO ] [ INFO ] stdev error: 0.0401951
[ INFO ] Utterance 6: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 4034.31ms [ INFO ] Utterance 6:
[ INFO ] Frames in utterance: 790 [ INFO ] Total time in Infer (HW and SW): 4034.31ms
[ INFO ] Average Infer time per frame: 5.11ms [ INFO ] Frames in utterance: 790
[ INFO ] [ INFO ] Average Infer time per frame: 5.11ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7648273 [ INFO ]
[ INFO ] avg error: 0.0482702 [ INFO ] max error: 0.7648273
[ INFO ] avg rms error: 0.0629734 [ INFO ] avg error: 0.0482702
[ INFO ] stdev error: 0.0404429 [ INFO ] avg rms error: 0.0629734
[ INFO ] [ INFO ] stdev error: 0.0404429
[ INFO ] Utterance 7: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 2854.04ms [ INFO ] Utterance 7:
[ INFO ] Frames in utterance: 622 [ INFO ] Total time in Infer (HW and SW): 2854.04ms
[ INFO ] Average Infer time per frame: 4.59ms [ INFO ] Frames in utterance: 622
[ INFO ] [ INFO ] Average Infer time per frame: 4.59ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.7389560 [ INFO ]
[ INFO ] avg error: 0.0465543 [ INFO ] max error: 0.7389560
[ INFO ] avg rms error: 0.0604941 [ INFO ] avg error: 0.0465543
[ INFO ] stdev error: 0.0386294 [ INFO ] avg rms error: 0.0604941
[ INFO ] [ INFO ] stdev error: 0.0386294
[ INFO ] Utterance 8: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 2493.28ms [ INFO ] Utterance 8:
[ INFO ] Frames in utterance: 548 [ INFO ] Total time in Infer (HW and SW): 2493.28ms
[ INFO ] Average Infer time per frame: 4.55ms [ INFO ] Frames in utterance: 548
[ INFO ] [ INFO ] Average Infer time per frame: 4.55ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.6680136 [ INFO ]
[ INFO ] avg error: 0.0439341 [ INFO ] max error: 0.6680136
[ INFO ] avg rms error: 0.0574614 [ INFO ] avg error: 0.0439341
[ INFO ] stdev error: 0.0370353 [ INFO ] avg rms error: 0.0574614
[ INFO ] [ INFO ] stdev error: 0.0370353
[ INFO ] Utterance 9: [ INFO ]
[ INFO ] Total time in Infer (HW and SW): 1654.67ms [ INFO ] Utterance 9:
[ INFO ] Frames in utterance: 368 [ INFO ] Total time in Infer (HW and SW): 1654.67ms
[ INFO ] Average Infer time per frame: 4.50ms [ INFO ] Frames in utterance: 368
[ INFO ] [ INFO ] Average Infer time per frame: 4.50ms
[ INFO ] Output blob name: affinetransform14 [ INFO ]
[ INFO ] Number scores per frame: 3425 [ INFO ] Output blob name: affinetransform14
[ INFO ] [ INFO ] Number scores per frame: 3425
[ INFO ] max error: 0.6550579 [ INFO ]
[ INFO ] avg error: 0.0467643 [ INFO ] max error: 0.6550579
[ INFO ] avg rms error: 0.0605045 [ INFO ] avg error: 0.0467643
[ INFO ] stdev error: 0.0383914 [ INFO ] avg rms error: 0.0605045
[ INFO ] [ INFO ] stdev error: 0.0383914
[ INFO ] Total sample time: 39722.60ms [ INFO ]
[ INFO ] File result.npz was created! [ INFO ] Total sample time: 39722.60ms
[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool [ INFO ] File result.npz was created!
``` [ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool
## See Also See Also
########
- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/integrate_with_your_application.md) - :doc:`Integrate the OpenVINO™ Runtime with Your Application <openvino_docs_OV_UG_Integrate_OV_with_your_application>`
- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - :doc:`Using OpenVINO™ Toolkit Samples <openvino_docs_OV_UG_Samples_Overview>`
- [Model Downloader](@ref omz_tools_downloader) - :doc:`Model Downloader <omz_tools_downloader>`
- [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) - :doc:`Model Optimizer <openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide>`
@endsphinxdirective
[openvino.runtime.Core.import_model]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.Core.html#openvino.runtime.Core.import_model
[openvino.runtime.CompiledModel.export_model]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.export_model
[openvino.runtime.Model.add_outputs]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.Model.html#openvino.runtime.Model.add_outputs
[openvino.runtime.set_batch]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.html#openvino.runtime.set_batch
[openvino.runtime.CompiledModel.inputs]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.inputs
[openvino.runtime.CompiledModel.outputs]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.outputs
[openvino.runtime.ConstOutput.any_name]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.ConstOutput.html#openvino.runtime.ConstOutput.any_name
[openvino.runtime.CompiledModel.create_infer_request]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.CompiledModel.html#openvino.runtime.CompiledModel.create_infer_request
[openvino.runtime.InferRequest.infer]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.infer
[openvino.runtime.InferRequest.model_outputs]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.model_outputs
[openvino.runtime.InferRequest.model_inputs]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.model_inputs
[openvino.runtime.InferRequest.query_state]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.query_state
[openvino.runtime.VariableState.reset]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.inference_engine.VariableState.html#openvino.inference_engine.VariableState.reset
[openvino.runtime.InferRequest.profiling_info]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.profiling_info
[openvino.runtime.ProfilingInfo.real_time]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.ProfilingInfo.html#openvino.runtime.ProfilingInfo.real_time
[openvino.runtime.InferRequest.get_input_tensor]:https://docs.openvino.ai/2022.3/api/ie_python_api/_autosummary/openvino.runtime.InferRequest.html#openvino.runtime.InferRequest.get_input_tensor