From 310eb81403736e10ecd075792363a785ad60e03f Mon Sep 17 00:00:00 2001 From: Dmitry Pigasin Date: Mon, 14 Feb 2022 19:03:19 +0300 Subject: [PATCH] [IE Samples] Update docs for C++ samples (#9937) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * update hello classification readme * update hello classification readme * update classification async readme * replace `network` with `model` * update example section with openvino-dev * update hello query device readme * Update hello reshape readme * Update ngraph func creation readme * update speech sample readme * update hello nv12 readme * Apply suggestions from code review review comments accepted Co-authored-by: Anastasiya Ageeva * Replace `Inference Engine` with `OpenVINO` * fix model ref * Replace `Inference Engine` by `OpenVINO™ Runtime` * Fix IR mentions Co-authored-by: Vladimir Dudnik Co-authored-by: Anastasiya Ageeva --- .../cpp/classification_sample_async/README.md | 198 ++++++++-------- .../classification_sample_async.h | 1 + .../cpp/classification_sample_async/main.cpp | 4 +- samples/cpp/hello_classification/README.md | 114 ++++++---- samples/cpp/hello_classification/main.cpp | 2 +- .../hello_nv12_input_classification/README.md | 117 +++++----- .../hello_nv12_input_classification/main.cpp | 4 +- samples/cpp/hello_query_device/README.md | 127 +++++------ samples/cpp/hello_reshape_ssd/README.md | 110 +++++---- samples/cpp/hello_reshape_ssd/main.cpp | 4 +- .../ngraph_function_creation_sample/README.md | 210 +++++++++-------- samples/cpp/speech_sample/README.md | 211 +++++++++--------- 12 files changed, 598 insertions(+), 504 deletions(-) diff --git a/samples/cpp/classification_sample_async/README.md b/samples/cpp/classification_sample_async/README.md index 8bb73cb95ce..c9f601cc4c8 100644 --- a/samples/cpp/classification_sample_async/README.md +++ b/samples/cpp/classification_sample_async/README.md @@ -1,33 +1,32 @@ # Image Classification Async C++ Sample {#openvino_inference_engine_samples_classification_sample_async_README} -This sample demonstrates how to execute an inference of image classification networks like AlexNet and GoogLeNet using Asynchronous Inference Request API. +This sample demonstrates how to do inference of image classification models using Asynchronous Inference Request API. +Models with only one input and output are supported. In addition to regular images, the sample also supports single-channel `ubyte` images as an input for LeNet model. -Image Classification Async C++ sample application demonstrates how to use the following Inference Engine C++ API in applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -|OpenVINO Runtime Version| `ov::get_openvino_version` | Get Openvino API version -|Available Devices| `ov::Core::get_available_devices`| Get version information of the devices for inference -| Asynchronous Infer | `ov::InferRequest::start_async`, `ov::InferRequest::set_callback` | Do asynchronous inference with callback -|Custom Extension Kernels| `ov::Core::add_extension`, `ov::Core::set_config`| Load extension library and config to the device -| Model Operations | `ov::set_batch`, `ov::get_batch` | Managing of model, operate with its batch size. Setting batch size using input image count. +| Feature | API | Description | +| :--- | :--- | :--- | +| Asynchronous Infer | `ov::InferRequest::start_async`, `ov::InferRequest::set_callback` | Do asynchronous inference with callback. | +| Model Operations | `ov::Output::get_shape`, `ov::set_batch` | Manage the model, operate with its batch size. Set batch size using input image count. | +| Infer Request Operations | `ov::InferRequest::get_input_tensor` | Get an input tensor. | +| Tensor Operations | `ov::shape_size`, `ov::Tensor::data` | Get a tensor shape size and its data. | -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). -| Options | Values | -|:--- |:--- -| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) -| Model Format | Inference Engine Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) -| Validated images | The sample uses OpenCV\* to [read input image](https://docs.opencv.org/master/d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56) (\*.bmp, \*.png), single-channel `ubyte` images. -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [Python](../../../samples/python/classification_sample_async/README.md) | +| Options | Values | +| :--- | :--- | +| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) | +| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) | +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [Python](../../../samples/python/classification_sample_async/README.md) | ## How It Works -Upon the start-up, the sample application reads command line parameters and loads specified network and input images (or a -folder with images) to the Inference Engine plugin. The batch size of the network is set according to the number of read images. The batch mode is an independent attribute on the asynchronous mode. Asynchronous mode works efficiently with any batch size. +At startup, the sample application reads command line parameters and loads the specified model and input images (or a +folder with images) to the OpenVINO™ Runtime plugin. The batch size of the model is set according to the number of read images. The batch mode is an independent attribute on the asynchronous mode. Asynchronous mode works efficiently with any batch size. Then, the sample creates an inference request object and assigns completion callback for it. In scope of the completion callback handling the inference request is executed again. @@ -37,14 +36,37 @@ After that, the application starts inference for the first infer request and wai When inference is done, the application outputs data to the standard output stream. You can place labels in .labels file near the model to get pretty output. You can see the explicit description of -each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running +Run the application with the `-h` option to see the usage instructions: + +``` +classification_sample_async -h +``` + +Usage instructions: + +``` +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... + +classification_sample_async [OPTION] +Options: + + -h Print usage instructions. + -m "" Required. Path to an .xml file with a trained model. + -i "" Required. Path to a folder with images or path to image files: a .ubyte file for LeNet and a .bmp file for other models. + -d "" Optional. Specify the target device to infer on (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:" format to specify the HETERO plugin. Sample will look for a suitable plugin for the device specified. + +Available target devices: +``` + To run the sample, you need specify a model and image: - you can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader). @@ -52,82 +74,72 @@ To run the sample, you need specify a model and image: > **NOTES**: > -> - By default, Inference Engine samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). +> - By default, OpenVINO™ Toolkit Samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). > -> - Before running the sample with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). +> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). > > - The sample accepts models in ONNX format (.onnx) that do not require preprocessing. -Running the application with the `-h` option yields the following usage message: - -``` -/classification_sample_async -h -InferenceEngine: - API version ............ - Build .................. - Description ....... API - -classification_sample_async [OPTION] -Options: - - -h Print a usage message. - -m "" Required. Path to an .xml file with a trained model. - -i "" Required. Path to a folder with images or path to an image files: a .ubyte file for LeNet and a .bmp file for the other networks. - -l "" Required for CPU plugin custom layers. Absolute path to a shared library with the kernels implementations - Or - -c "" Required for GPU, MYRIAD, HDDL custom kernels. Absolute path to the .xml config file with the kernels descriptions. - -d "" Optional. Specify the target device to infer on (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:" format to specify HETERO plugin. Sample will look for a suitable plugin for device specified. - -nt "" Optional. Number of top results. Default value is 10. - - Available target devices: - -``` - -Running the application with the empty list of options yields the usage message given above and an error message. - ### Example -1. Download a pre-trained model using [Model Downloader](@ref omz_tools_downloader): -``` -python /downloader.py --name alexnet -``` -2. If a model is not in the Inference Engine IR or ONNX format, it must be converted. You can do this using the model converter script: +1. Install the `openvino-dev` Python package to use Open Model Zoo Tools: ``` -python /converter.py --name alexnet +python -m pip install openvino-dev[caffe,onnx,tensorflow2,pytorch,mxnet] ``` -3. Perform inference of `car.bmp` using `alexnet` model on a `GPU`, for example: +2. Download a pre-trained model using: ``` -/classification_sample_async -m /alexnet.xml -i /car.bmp -d GPU +omz_downloader --name googlenet-v1 +``` + +3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter: + +``` +omz_converter --name googlenet-v1 +``` + +4. Perform inference of `dog.bmp` using `googlenet-v1` model on a `GPU`, for example: + +``` +classification_sample_async -m googlenet-v1.xml -i dog.bmp -d GPU ``` ## Sample Output -By default the application outputs top-10 inference results for each infer request. - ``` -[ INFO ] InferenceEngine: - IE version ......... 2021.4.0 - Build ........... 2021.4.0-3839-cd81789d294-releases/2021/4 +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] [ INFO ] Parsing input parameters [ INFO ] Files were added: 1 -[ INFO ] C:\images\car.bmp -[ INFO ] Loading Inference Engine -[ INFO ] Device info: - GPU - clDNNPlugin version ......... 2021.4.0 - Build ........... 2021.4.0-3839-cd81789d294-releases/2021/4 - -[ INFO ] Loading network files: -[ INFO ] C:\openvino\deployment_tools\open_model_zoo\tools\downloader\public\alexnet\FP32\alexnet.xml -[ INFO ] Preparing input blobs -[ WARNING ] Image is resized from (749, 637) to (227, 227) -[ INFO ] Batch size is 1 -[ INFO ] Loading model to the device +[ INFO ] /images/dog.bmp +[ INFO ] Loading model files: +[ INFO ] /models/googlenet-v1.xml +[ INFO ] model name: GoogleNet +[ INFO ] inputs +[ INFO ] input name: data +[ INFO ] input type: f32 +[ INFO ] input shape: {1, 3, 224, 224} +[ INFO ] outputs +[ INFO ] output name: prob +[ INFO ] output type: f32 +[ INFO ] output shape: {1, 1000} +[ INFO ] Read input images +[ INFO ] Set batch size 1 +[ INFO ] model name: GoogleNet +[ INFO ] inputs +[ INFO ] input name: data +[ INFO ] input type: u8 +[ INFO ] input shape: {1, 224, 224, 3} +[ INFO ] outputs +[ INFO ] output name: prob +[ INFO ] output type: f32 +[ INFO ] output shape: {1, 1000} +[ INFO ] Loading model to the device GPU [ INFO ] Create infer request -[ INFO ] Start inference (10 asynchronous executions) +[ INFO ] Start inference (asynchronous executions) [ INFO ] Completed 1 async request execution [ INFO ] Completed 2 async request execution [ INFO ] Completed 3 async request execution @@ -138,33 +150,29 @@ By default the application outputs top-10 inference results for each infer reque [ INFO ] Completed 8 async request execution [ INFO ] Completed 9 async request execution [ INFO ] Completed 10 async request execution -[ INFO ] Processing output blobs +[ INFO ] Completed async requests execution Top 10 results: -Image C:\images\car.bmp +Image /images/dog.bmp classid probability ------- ----------- -656 0.6645315 -654 0.1121185 -581 0.0698451 -874 0.0334973 -436 0.0259718 -817 0.0173190 -675 0.0109321 -511 0.0109075 -569 0.0083093 -717 0.0063173 - -[ INFO ] Execution successful - -[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool +156 0.8935547 +218 0.0608215 +215 0.0217133 +219 0.0105667 +212 0.0018835 +217 0.0018730 +152 0.0018730 +157 0.0015745 +154 0.0012817 +220 0.0010099 ``` ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Downloader](@ref omz_tools_downloader) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) diff --git a/samples/cpp/classification_sample_async/classification_sample_async.h b/samples/cpp/classification_sample_async/classification_sample_async.h index b0325e115df..0a474ecfa52 100644 --- a/samples/cpp/classification_sample_async/classification_sample_async.h +++ b/samples/cpp/classification_sample_async/classification_sample_async.h @@ -52,5 +52,6 @@ static void show_usage() { std::cout << std::endl; std::cout << " -h " << help_message << std::endl; std::cout << " -m \"\" " << model_message << std::endl; + std::cout << " -i \"\" " << image_message << std::endl; std::cout << " -d \"\" " << target_device_message << std::endl; } diff --git a/samples/cpp/classification_sample_async/main.cpp b/samples/cpp/classification_sample_async/main.cpp index 6c052d5fe01..fe2684f273d 100644 --- a/samples/cpp/classification_sample_async/main.cpp +++ b/samples/cpp/classification_sample_async/main.cpp @@ -87,8 +87,8 @@ int main(int argc, char* argv[]) { std::shared_ptr model = core.read_model(FLAGS_m); printInputAndOutputsInfo(*model); - OPENVINO_ASSERT(model->get_parameters().size() == 1, "Sample supports models with 1 input only"); - OPENVINO_ASSERT(model->get_results().size() == 1, "Sample supports models with 1 output only"); + OPENVINO_ASSERT(model->inputs().size() == 1, "Sample supports models with 1 input only"); + OPENVINO_ASSERT(model->outputs().size() == 1, "Sample supports models with 1 output only"); // -------- Step 3. Configure preprocessing -------- const ov::Layout tensor_layout{"NHWC"}; diff --git a/samples/cpp/hello_classification/README.md b/samples/cpp/hello_classification/README.md index b5e53a4fc49..27af315a355 100644 --- a/samples/cpp/hello_classification/README.md +++ b/samples/cpp/hello_classification/README.md @@ -1,39 +1,43 @@ # Hello Classification C++ Sample {#openvino_inference_engine_samples_hello_classification_README} -This sample demonstrates how to execute an inference of image classification networks like AlexNet and GoogLeNet using Synchronous Inference Request API, input auto-resize feature and support of UNICODE paths. +This sample demonstrates how to do inference of image classification models using Synchronous Inference Request API. +Models with only one input and output are supported. -Hello Classification C++ sample application demonstrates how to use the following Inference Engine C++ API in applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -| Basic Infer Flow | `ov::Core::read_model`, `ov::Core::compile_model`, `ov::CompiledModel::create_infer_request`, `ov::InferRequest::get_input_tensor`, `ov::InferRequest::set_input_tensor`, `ov::InferRequest::get_output_tensor` | Common API to do inference: configure input and output tensors, reading model, create infer request -| Synchronous Infer | `ov::InferRequest::infer` | Do synchronous inference -| Model Operations | `ov::Model::inputs`, `ov::Model::outputs` | Managing of model -| Tensor Operations| `ov::Tensor::get_element_type`, `ov::Tensor::get_shape`, `ov::Tensor::data` | Work with storing inputs, outputs of the model, weights and biases of the layers -| Input auto-resize | `ov::preprocess::PreProcessSteps::resize`, `ov::preprocess::InputInfo::model::set_layout` | Set image of the original size as input for a model with other input size. Resize and layout conversions will be performed automatically by the corresponding plugin just before inference +| Feature | API | Description | +| :--- | :--- | :--- | +| OpenVINO Runtime Version | `ov::get_openvino_version` | Get Openvino API version | +| Basic Infer Flow | `ov::Core::read_model`, `ov::Core::compile_model`, `ov::CompiledModel::create_infer_request`, `ov::InferRequest::set_input_tensor`, `ov::InferRequest::get_output_tensor` | Common API to do inference: read and compile a model, create an infer request, configure input and output tensors | +| Synchronous Infer | `ov::InferRequest::infer` | Do synchronous inference | +| Model Operations | `ov::Model::inputs`, `ov::Model::outputs` | Get inputs and outputs of a model | +| Tensor Operations | `ov::Tensor::get_shape` | Get a tensor shape | +| Preprocessing | `ov::preprocess::InputTensorInfo::set_element_type`, `ov::preprocess::InputTensorInfo::set_layout`, `ov::preprocess::InputTensorInfo::set_spatial_static_shape`, `ov::preprocess::PreProcessSteps::resize`, `ov::preprocess::InputModelInfo::set_layout`, `ov::preprocess::OutputTensorInfo::set_element_type`, `ov::preprocess::PrePostProcessor::build` | Set image of the original size as input for a model with other input size. Resize and layout conversions are performed automatically by the corresponding plugin just before inference. | -| Options | Values | -|:--- |:--- -| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) -| Model Format | Inference Engine Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) -| Validated images | The sample uses OpenCV\* to [read input image](https://docs.opencv.org/master/d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56) (\*.bmp, \*.png) -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [C](../../../samples/c/hello_classification/README.md), [Python](../../../samples/python/hello_classification/README.md) | +| Options | Values | +| :--- | :--- | +| Validated Models | [alexnet](@ref omz_models_model_alexnet), [googlenet-v1](@ref omz_models_model_googlenet_v1) | +| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) | +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [C](../../../samples/c/hello_classification/README.md), [Python](../../../samples/python/hello_classification/README.md) | ## How It Works -Upon the start-up, the sample application reads command line parameters, loads specified network and an image to the Inference Engine plugin. -Then, the sample creates an synchronous inference request object. When inference is done, the application outputs data to the standard output stream. +At startup, the sample application reads command line parameters, prepares input data, loads a specified model and image to the OpenVINO™ Runtime plugin and performs synchronous inference. Then processes output data and write it to a standard output stream. You can see the explicit description of -each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running +``` +hello_classification +``` + To run the sample, you need specify a model and image: - you can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader). @@ -41,28 +45,36 @@ To run the sample, you need specify a model and image: > **NOTES**: > -> - By default, Inference Engine samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). +> - By default, OpenVINO™ Toolkit Samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). > -> - Before running the sample with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). +> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). > > - The sample accepts models in ONNX format (.onnx) that do not require preprocessing. ### Example -1. Download a pre-trained model using [Model Downloader](@ref omz_tools_downloader): -``` -python /downloader.py --name alexnet -``` -2. If a model is not in the Inference Engine IR or ONNX format, it must be converted. You can do this using the model converter script: +1. Install the `openvino-dev` Python package to use Open Model Zoo Tools: ``` -python /converter.py --name alexnet +python -m pip install openvino-dev[caffe,onnx,tensorflow2,pytorch,mxnet] ``` -3. Perform inference of `car.bmp` using `alexnet` model on a `GPU`, for example: +2. Download a pre-trained model using: ``` -/hello_classification /alexnet.xml /car.bmp GPU +omz_downloader --name googlenet-v1 +``` + +3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter: + +``` +omz_converter --name googlenet-v1 +``` + +4. Perform inference of `car.bmp` using the `googlenet-v1` model on a `GPU`, for example: + +``` +hello_classification googlenet-v1.xml car.bmp GPU ``` ## Sample Output @@ -70,29 +82,41 @@ python /converter.py --name alexnet The application outputs top-10 inference results. ``` +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] +[ INFO ] Loading model files: /models/googlenet-v1.xml +[ INFO ] model name: GoogleNet +[ INFO ] inputs +[ INFO ] input name: data +[ INFO ] input type: f32 +[ INFO ] input shape: {1, 3, 224, 224} +[ INFO ] outputs +[ INFO ] output name: prob +[ INFO ] output type: f32 +[ INFO ] output shape: {1, 1000} + Top 10 results: -Image /opt/intel/openvino/samples/scripts/car.png +Image /images/car.bmp classid probability ------- ----------- -656 0.6664789 -654 0.1129405 -581 0.0684867 -874 0.0333845 -436 0.0261321 -817 0.0167310 -675 0.0109796 -511 0.0105919 -569 0.0081782 -717 0.0063356 - -This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool +656 0.8139648 +654 0.0550537 +468 0.0178375 +436 0.0165405 +705 0.0111694 +817 0.0105820 +581 0.0086823 +575 0.0077515 +734 0.0064468 +785 0.0043983 ``` ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Downloader](@ref omz_tools_downloader) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) diff --git a/samples/cpp/hello_classification/main.cpp b/samples/cpp/hello_classification/main.cpp index b5cbe3ec1c5..148e02dfdb8 100644 --- a/samples/cpp/hello_classification/main.cpp +++ b/samples/cpp/hello_classification/main.cpp @@ -28,7 +28,7 @@ int tmain(int argc, tchar* argv[]) { // -------- Parsing and validation of input arguments -------- if (argc != 4) { - slog::info << "Usage : " << argc << " " << slog::endl; + slog::info << "Usage : " << argv[0] << " " << slog::endl; return EXIT_FAILURE; } diff --git a/samples/cpp/hello_nv12_input_classification/README.md b/samples/cpp/hello_nv12_input_classification/README.md index 0a6611fb427..83fd42793a0 100644 --- a/samples/cpp/hello_nv12_input_classification/README.md +++ b/samples/cpp/hello_nv12_input_classification/README.md @@ -1,40 +1,42 @@ # Hello NV12 Input Classification C++ Sample {#openvino_inference_engine_samples_hello_nv12_input_classification_README} -This sample demonstrates how to execute an inference of image classification networks like AlexNet with images in NV12 color format using Synchronous Inference Request API and [input reshape feature](../../../docs/OV_Runtime_UG/ShapeInference.md). +This sample demonstrates how to execute an inference of image classification models with images in NV12 color format using Synchronous Inference Request API. -Hello NV12 Input Classification C++ Sample demonstrates how to use the NV12 automatic input pre-processing API of the Inference Engine in your applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -|OpenVINO Runtime Core Operations| `ov::Core::get_metric` | Gets general runtime metric for dedicated hardware -| Tensor Operations | `ov::Tensor::get_element_type`, `ov::Tensor::get_shape`, `ov::Tensor::data` | Work with storing inputs, outputs of the model, weights and biases of the layers -| Input in N12 color format | `ov::preprocess::InputTensorInfo::set_color_format` | Change the color format of the input data -| Model Input Reshape | `ov::Model::get_output_shape`, `ov::Model::reshape`, `ov::get_batch` | Set the batch size equal to the number of input images +| Feature | API | Description | +| :--- | :--- | :--- | +| Node Operations | `ov::Output::get_any_name` | Get a layer name | +| Infer Request Operations | `ov::InferRequest::set_tensor`, `ov::InferRequest::get_tensor` | Operate with tensors | +| Preprocessing | `ov::preprocess::InputTensorInfo::set_color_format`, `ov::preprocess::PreProcessSteps::convert_element_type`, `ov::preprocess::PreProcessSteps::convert_color` | Change the color format of the input data | -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). -| Options | Values | -|:--- |:--- -| Validated Models | [alexnet](@ref omz_models_model_alexnet) -| Model Format | Inference Engine Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) -| Validated images | An uncompressed image in the NV12 color format - \*.yuv -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [C](../../../samples/c/hello_nv12_input_classification/README.md) | +| Options | Values | +| :--- |:--- | +| Validated Models | [alexnet](@ref omz_models_model_alexnet) | +| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) | +| Validated images | An uncompressed image in the NV12 color format - \*.yuv | +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [C](../../../samples/c/hello_nv12_input_classification/README.md) | ## How It Works -Upon the start-up, the sample application reads command-line parameters, loads specified network and an -image in the NV12 color format to an Inference Engine plugin. Then, the sample creates an synchronous inference request object. When inference is done, the application outputs data to the standard output stream. You can place labels in .labels file near the model to get pretty output. +At startup, the sample application reads command line parameters, loads the specified model and an +image in the NV12 color format to an OpenVINO™ Runtime plugin. Then, the sample creates an synchronous inference request object. When inference is done, the application outputs data to the standard output stream. You can place labels in .labels file near the model to get pretty output. -You can see the explicit description of each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +You can see the explicit description of each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. - +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running +``` +hello_nv12_input_classification +``` + To run the sample, you need specify a model and image: - you can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader). @@ -46,7 +48,7 @@ as FFmpeg\* or GStreamer\*. The following command shows how to convert an ordina uncompressed NV12 image using FFmpeg: ```sh -ffmpeg -i cat.jpg -pix_fmt nv12 cat.yuv +ffmpeg -i cat.jpg -pix_fmt nv12 car.yuv ``` > **NOTES**: @@ -55,31 +57,38 @@ ffmpeg -i cat.jpg -pix_fmt nv12 cat.yuv > image path. The sample expects the logical size of the image, not the buffer size. For example, > for 640x480 BGR/RGB image the corresponding NV12 logical image size is also 640x480, whereas the > buffer size is 640x720. -> - By default, this sample expects that network input has BGR channels order. If you trained your +> - By default, this sample expects that model input has BGR channels order. If you trained your > model to work with RGB order, you need to reconvert your model using the Model Optimizer tool > with `--reverse_input_channels` argument specified. For more information about the argument, > refer to **When to Reverse Input Channels** section of > [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). -> - Before running the sample with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). +> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). > > - The sample accepts models in ONNX format (.onnx) that do not require preprocessing. ### Example -1. Download a pre-trained model using [Model Downloader](@ref omz_tools_downloader): -``` -python /downloader.py --name alexnet -``` -2. If a model is not in the Inference Engine IR or ONNX format, it must be converted. You can do this using the model converter script: +1. Install openvino-dev python package if you don't have it to use Open Model Zoo Tools: ``` -python /converter.py --name alexnet +python -m pip install openvino-dev[caffe,onnx,tensorflow2,pytorch,mxnet] ``` -3. Perform inference of NV12 image using `alexnet` model on a `CPU`, for example: +2. Download a pre-trained model: +``` +omz_downloader --name alexnet +``` + +3. If a model is not in the IR or ONNX format, it must be converted. You can do this using the model converter: ``` -/hello_nv12_input_classification /alexnet.xml /cat.yuv 300x300 CPU +omz_converter --name alexnet +``` + +4. Perform inference of NV12 image using `alexnet` model on a `CPU`, for example: + +``` +hello_nv12_input_classification alexnet.xml car.yuv 300x300 CPU ``` ## Sample Output @@ -87,33 +96,41 @@ python /converter.py --name alexnet The application outputs top-10 inference results. ``` -[ INFO ] Files were added: 1 -[ INFO ] ./cat.yuv -Batch size is 1 +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] +[ INFO ] Loading model files: \models\alexnet.xml +[ INFO ] model name: AlexNet +[ INFO ] inputs +[ INFO ] input name: data +[ INFO ] input type: f32 +[ INFO ] input shape: {1, 3, 227, 227} +[ INFO ] outputs +[ INFO ] output name: prob +[ INFO ] output type: f32 +[ INFO ] output shape: {1, 1000} Top 10 results: -Image ./cat.yuv +Image \images\car.yuv classid probability ------- ----------- -435 0.0917327 -876 0.0817254 -999 0.0693054 -587 0.0437265 -666 0.0389570 -419 0.0328923 -285 0.0303094 -700 0.0299405 -696 0.0216280 -855 0.0203389 - -This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool +656 0.6668988 +654 0.1125269 +581 0.0679280 +874 0.0340229 +436 0.0257744 +817 0.0169367 +675 0.0110199 +511 0.0106134 +569 0.0083373 +717 0.0061734 ``` ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Downloader](@ref omz_tools_downloader) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) diff --git a/samples/cpp/hello_nv12_input_classification/main.cpp b/samples/cpp/hello_nv12_input_classification/main.cpp index 4798b8451c0..9ec7bd32f37 100644 --- a/samples/cpp/hello_nv12_input_classification/main.cpp +++ b/samples/cpp/hello_nv12_input_classification/main.cpp @@ -104,8 +104,8 @@ int main(int argc, char* argv[]) { std::shared_ptr model = core.read_model(model_path); printInputAndOutputsInfo(*model); - OPENVINO_ASSERT(model->get_parameters().size() == 1, "Sample supports models with 1 input only"); - OPENVINO_ASSERT(model->get_results().size() == 1, "Sample supports models with 1 output only"); + OPENVINO_ASSERT(model->inputs().size() == 1, "Sample supports models with 1 input only"); + OPENVINO_ASSERT(model->outputs().size() == 1, "Sample supports models with 1 output only"); std::string input_tensor_name = model->input().get_any_name(); std::string output_tensor_name = model->output().get_any_name(); diff --git a/samples/cpp/hello_query_device/README.md b/samples/cpp/hello_query_device/README.md index 1f1a6a7d4b8..8b9ee2b12d6 100644 --- a/samples/cpp/hello_query_device/README.md +++ b/samples/cpp/hello_query_device/README.md @@ -1,35 +1,34 @@ # Hello Query Device C++ Sample {#openvino_inference_engine_samples_hello_query_device_README} -This sample demonstrates how to execute an query Inference Engine devices, prints their metrics and default configuration values, using [Query Device API feature](../../../docs/OV_Runtime_UG/InferenceEngine_QueryAPI.md). +This sample demonstrates how to execute an query OpenVINO™ Runtime devices, prints their metrics and default configuration values, using [Query Device API feature](../../../docs/OV_Runtime_UG/InferenceEngine_QueryAPI.md). -Hello Query Device C++ sample application demonstrates how to use the following Inference Engine C++ API in applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -|Available Devices|`ov::Core::get_available_devices`, `ov::Core::get_metric`, `ov::Core::get_config`| Get available devices information and configuration for inference +| Feature | API | Description | +| :--- | :--- | :--- | +| Available Devices | `ov::Core::get_available_devices`, `ov::Core::get_property` | Get available devices information and configuration for inference | -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). | Options | Values | -|:--- |:--- -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [Python](../../../samples/python/hello_query_device/README.md) | +| :--- |:--- +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [Python](../../../samples/python/hello_query_device/README.md) | ## How It Works -The sample queries all available Inference Engine devices, prints their supported metrics and plugin configuration parameters. +The sample queries all available OpenVINO™ Runtime devices, prints their supported metrics and plugin configuration parameters. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running To see quired information, run the following: ``` -/hello_query_device -h -Usage : hello_query_device +hello_query_device ``` ## Sample Output @@ -37,57 +36,59 @@ Usage : hello_query_device The application prints all available devices with their supported metrics and default values for configuration parameters: ``` -Available devices: - Device: CPU - Metrics: - SUPPORTED_METRICS : [ AVAILABLE_DEVICES SUPPORTED_METRICS FULL_DEVICE_NAME OPTIMIZATION_CAPABILITIES SUPPORTED_CONFIG_KEYS RANGE_FOR_ASYNC_INFER_REQUESTS RANGE_FOR_STREAMS ] - FULL_DEVICE_NAME : Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz - OPTIMIZATION_CAPABILITIES : [ FP32 FP16 INT8 BIN ] - SUPPORTED_CONFIG_KEYS : [ CPU_BIND_THREAD CPU_THREADS_NUM CPU_THROUGHPUT_STREAMS DUMP_EXEC_GRAPH_AS_DOT DYN_BATCH_ENABLED DYN_BATCH_LIMIT ENFORCE_BF16 EXCLUSIVE_ASYNC_REQUESTS PERF_COUNT ] - RANGE_FOR_ASYNC_INFER_REQUESTS : { 1, 1, 1 } - RANGE_FOR_STREAMS : { 1, 8 } - Default values for device configuration keys: - CPU_BIND_THREAD : NUMA - CPU_THREADS_NUM : 0 - CPU_THROUGHPUT_STREAMS : 1 - DUMP_EXEC_GRAPH_AS_DOT : "" - DYN_BATCH_ENABLED : NO - DYN_BATCH_LIMIT : 0 - ENFORCE_BF16 : NO - EXCLUSIVE_ASYNC_REQUESTS : NO - PERF_COUNT : NO - - Device: GPU - Metrics: - SUPPORTED_METRICS : [ AVAILABLE_DEVICES SUPPORTED_METRICS FULL_DEVICE_NAME OPTIMIZATION_CAPABILITIES SUPPORTED_CONFIG_KEYS RANGE_FOR_ASYNC_INFER_REQUESTS RANGE_FOR_STREAMS ] - FULL_DEVICE_NAME : Intel(R) UHD Graphics 620 (iGPU) - OPTIMIZATION_CAPABILITIES : [ FP32 BIN FP16 ] - SUPPORTED_CONFIG_KEYS : [ CACHE_DIR CLDNN_ENABLE_FP16_FOR_QUANTIZED_MODELS CLDNN_GRAPH_DUMPS_DIR GPU_MAX_NUM_THREADS CLDNN_MEM_POOL CLDNN_NV12_TWO_INPUTS CLDNN_PLUGIN_PRIORITY CLDNN_PLUGIN_THROTTLE CLDNN_SOURCES_DUMPS_DIR GPU_ENABLE_LOOP_UNROLLING CONFIG_FILE DEVICE_ID DUMP_KERNELS DYN_BATCH_ENABLED EXCLUSIVE_ASYNC_REQUESTS GPU_THROUGHPUT_STREAMS PERF_COUNT TUNING_FILE TUNING_MODE ] - RANGE_FOR_ASYNC_INFER_REQUESTS : { 1, 2, 1 } - RANGE_FOR_STREAMS : { 1, 2 } - Default values for device configuration keys: - CACHE_DIR : "" - CLDNN_ENABLE_FP16_FOR_QUANTIZED_MODELS : YES - CLDNN_GRAPH_DUMPS_DIR : "" - CLDNN_MEM_POOL : YES - CLDNN_NV12_TWO_INPUTS : NO - CLDNN_PLUGIN_PRIORITY : 0 - CLDNN_PLUGIN_THROTTLE : 0 - CLDNN_SOURCES_DUMPS_DIR : "" - GPU_MAX_NUM_THREADS : 8 - GPU_ENABLE_LOOP_UNROLLING : YES - CONFIG_FILE : "" - DEVICE_ID : "" - DUMP_KERNELS : NO - DYN_BATCH_ENABLED : NO - EXCLUSIVE_ASYNC_REQUESTS : NO - GPU_THROUGHPUT_STREAMS : 1 - PERF_COUNT : NO - TUNING_FILE : "" - TUNING_MODE : TUNING_DISABLED +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] +[ INFO ] Available devices: +[ INFO ] CPU +[ INFO ] SUPPORTED_METRICS: +[ INFO ] AVAILABLE_DEVICES : [ ] +[ INFO ] FULL_DEVICE_NAME : Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz +[ INFO ] OPTIMIZATION_CAPABILITIES : [ FP32 FP16 INT8 BIN ] +[ INFO ] RANGE_FOR_ASYNC_INFER_REQUESTS : { 1, 1, 1 } +[ INFO ] RANGE_FOR_STREAMS : { 1, 8 } +[ INFO ] IMPORT_EXPORT_SUPPORT : true +[ INFO ] SUPPORTED_CONFIG_KEYS (default values): +[ INFO ] CACHE_DIR : "" +[ INFO ] CPU_BIND_THREAD : NO +[ INFO ] CPU_THREADS_NUM : 0 +[ INFO ] CPU_THROUGHPUT_STREAMS : 1 +[ INFO ] DUMP_EXEC_GRAPH_AS_DOT : "" +[ INFO ] DYN_BATCH_ENABLED : NO +[ INFO ] DYN_BATCH_LIMIT : 0 +[ INFO ] ENFORCE_BF16 : NO +[ INFO ] EXCLUSIVE_ASYNC_REQUESTS : NO +[ INFO ] PERFORMANCE_HINT : "" +[ INFO ] PERFORMANCE_HINT_NUM_REQUESTS : 0 +[ INFO ] PERF_COUNT : NO +[ INFO ] +[ INFO ] GNA +[ INFO ] SUPPORTED_METRICS: +[ INFO ] AVAILABLE_DEVICES : [ GNA_SW_EXACT ] +[ INFO ] OPTIMAL_NUMBER_OF_INFER_REQUESTS : 1 +[ INFO ] FULL_DEVICE_NAME : GNA_SW_EXACT +[ INFO ] GNA_LIBRARY_FULL_VERSION : 3.0.0.1455 +[ INFO ] IMPORT_EXPORT_SUPPORT : true +[ INFO ] SUPPORTED_CONFIG_KEYS (default values): +[ INFO ] EXCLUSIVE_ASYNC_REQUESTS : NO +[ INFO ] GNA_COMPACT_MODE : YES +[ INFO ] GNA_COMPILE_TARGET : "" +[ INFO ] GNA_DEVICE_MODE : GNA_SW_EXACT +[ INFO ] GNA_EXEC_TARGET : "" +[ INFO ] GNA_FIRMWARE_MODEL_IMAGE : "" +[ INFO ] GNA_FIRMWARE_MODEL_IMAGE_GENERATION : "" +[ INFO ] GNA_LIB_N_THREADS : 1 +[ INFO ] GNA_PRECISION : I16 +[ INFO ] GNA_PWL_MAX_ERROR_PERCENT : 1.000000 +[ INFO ] GNA_PWL_UNIFORM_DESIGN : NO +[ INFO ] GNA_SCALE_FACTOR : 1.000000 +[ INFO ] GNA_SCALE_FACTOR_0 : 1.000000 +[ INFO ] LOG_LEVEL : LOG_NONE +[ INFO ] PERF_COUNT : NO +[ INFO ] SINGLE_THREAD : YES ``` ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) diff --git a/samples/cpp/hello_reshape_ssd/README.md b/samples/cpp/hello_reshape_ssd/README.md index 777bb9cdafc..ef9d21ba169 100644 --- a/samples/cpp/hello_reshape_ssd/README.md +++ b/samples/cpp/hello_reshape_ssd/README.md @@ -1,24 +1,25 @@ # Hello Reshape SSD C++ Sample {#openvino_inference_engine_samples_hello_reshape_ssd_README} -This sample demonstrates how to execute an inference of object detection networks like SSD-VGG using Synchronous Inference Request API, [input reshape feature](../../../docs/OV_Runtime_UG/ShapeInference.md). +This sample demonstrates how to do synchronous inference of object detection models using [input reshape feature](../../../docs/OV_Runtime_UG/ShapeInference.md). +Models with only one input and output are supported. -Hello Reshape SSD C++ sample application demonstrates how to use the following Inference Engine C++ API in applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -| Model Operations | `ov::Core::read_model`, `ov::Core::compile_model` | Managing of model -| Model Input Reshape | `ov::Model::reshape`| Resize model to match image sizes and given batch -| Tensor Operations | `ov::Tensor::get_element_type`, `ov::Tensor::get_shape`, `ov::Tensor::data` | Work with storing inputs, outputs of the model, weights and biases of the layers +| Feature | API | Description | +| :--- | :--- | :--- | +| Node operations | `ov::Node::get_type_info`, `ngraph::op::DetectionOutput::get_type_info_static`, `ov::Output::get_any_name`, `ov::Output::get_shape` | Get a node info | +| Model Operations | `ov::Model::get_ops`, `ov::Model::reshape` | Get model nodes, reshape input | +| Tensor Operations | `ov::Tensor::data` | Get a tensor data | +| Preprocessing | `ov::preprocess::PreProcessSteps::convert_element_type`, `ov::preprocess::PreProcessSteps::convert_layout` | Model input preprocessing | -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). -| Options | Values | -|:--- |:--- -| Validated Models | [person-detection-retail-0013](@ref omz_models_model_person_detection_retail_0013) -| Model Format | Inference Engine Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) -| Validated images | The sample uses OpenCV\* to [read input image](https://docs.opencv.org/master/d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56) (\*.bmp, \*.png) -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [Python](../../../samples/python/hello_reshape_ssd/README.md) | +| Options | Values | +| :--- | :--- | +| Validated Models | [person-detection-retail-0013](@ref omz_models_model_person_detection_retail_0013) | +| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin), ONNX (\*.onnx) | +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [Python](../../../samples/python/hello_reshape_ssd/README.md) | ## How It Works @@ -26,14 +27,18 @@ Upon the start-up the sample application reads command line parameters, loads sp Engine plugin. Then, the sample creates an synchronous inference request object. When inference is done, the application creates output image and output data to the standard output stream. You can see the explicit description of -each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running +``` +hello_reshape_ssd +``` + To run the sample, you need specify a model and image: - you can use [public](@ref omz_models_group_public) or [Intel's](@ref omz_models_group_intel) pre-trained models from the Open Model Zoo. The models can be downloaded using the [Model Downloader](@ref omz_tools_downloader). @@ -41,34 +46,36 @@ To run the sample, you need specify a model and image: > **NOTES**: > -> - By default, Inference Engine samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). +> - By default, OpenVINO™ Toolkit Samples and demos expect input with BGR channels order. If you trained your model to work with RGB order, you need to manually rearrange the default channels order in the sample or demo application or reconvert your model using the Model Optimizer tool with `--reverse_input_channels` argument specified. For more information about the argument, refer to **When to Reverse Input Channels** section of [Converting a Model](../../../docs/MO_DG/prepare_model/convert_model/Converting_Model.md). > -> - Before running the sample with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). +> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). > > - The sample accepts models in ONNX format (\*.onnx) that do not require preprocessing. -You can use the following command to do inference on CPU of an image using a trained SSD network: - -``` -/hello_reshape_ssd -``` - ### Example -1. Download a pre-trained model using [Model Downloader](@ref omz_tools_downloader): -``` -python /downloader.py --name person-detection-retail-0013 -``` -2. `person-detection-retail-0013` model does not need to be converted, because it is already in necessary format, so you can skip this step. If you want to use a other model that is not in the Inference Engine IR or ONNX format, you can convert it using the model converter script: +1. Install openvino-dev python package if you don't have it to use Open Model Zoo Tools: ``` -python /converter.py --name +python -m pip install openvino-dev[caffe,onnx,tensorflow2,pytorch,mxnet] ``` -3. Perform inference of `person_detection.png` using `person-detection-retail-0013` model on a `GPU`, for example: +2. Download a pre-trained model using: ``` -/hello_reshape_ssd /person-detection-retail-0013.xml /person_detection.png GPU 1 +omz_downloader --name person-detection-retail-0013 +``` + +3. `person-detection-retail-0013` does not need to be converted, because it is already in necessary format, so you can skip this step. If you want to use another model that is not in the IR or ONNX format, you can convert it using the model converter script: + +``` +omz_converter --name +``` + +4. Perform inference of `person_detection.bmp` using `person-detection-retail-0013` model on a `GPU`, for example: + +``` +hello_reshape_ssd person-detection-retail-0013.xml person_detection.bmp GPU ``` ## Sample Output @@ -78,33 +85,38 @@ of the detected objects along with the respective confidence values and the coor rectangles to the standard output stream. ``` -[ INFO ] Loading model files: C:\temp\models\public\ssd_mobilenet_v1_fpn_coco\FP16\ssd_mobilenet_v1_fpn_coco.xml -[ INFO ] model name: ssd_mobilenet_v1_fpn_coco +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] +[ INFO ] Loading model files: \models\person-detection-retail-0013.xml +[ INFO ] model name: ResMobNet_v4 (LReLU) with single SSD head [ INFO ] inputs -[ INFO ] input name: image_tensor +[ INFO ] input name: data [ INFO ] input type: f32 -[ INFO ] input shape: {1, 3, 640, 640} +[ INFO ] input shape: {1, 3, 320, 544} [ INFO ] outputs -[ INFO ] output name: DetectionOutput +[ INFO ] output name: detection_out [ INFO ] output type: f32 -[ INFO ] output shape: {1, 1, 100, 7} -Reshape network to the image size = [512x512] with batch = 1 -[ INFO ] model name: ssd_mobilenet_v1_fpn_coco +[ INFO ] output shape: {1, 1, 200, 7} +Reshape network to the image size = [960x1699] +[ INFO ] model name: ResMobNet_v4 (LReLU) with single SSD head [ INFO ] inputs -[ INFO ] input name: image_tensor +[ INFO ] input name: data [ INFO ] input type: f32 -[ INFO ] input shape: {1, 3, 512, 512} +[ INFO ] input shape: {1, 3, 960, 1699} [ INFO ] outputs -[ INFO ] output name: DetectionOutput +[ INFO ] output name: detection_out [ INFO ] output type: f32 -[ INFO ] output shape: {1, 1, 100, 7} -[0,18] element, prob = 0.781129 (109,52)-(342,441) batch id = 0 -The resulting image was saved in the file: hello_reshape_ssd_batch_0.bmp +[ INFO ] output shape: {1, 1, 200, 7} +[0,1] element, prob = 0.716309, (852,187)-(983,520) +The resulting image was saved in the file: hello_reshape_ssd_output.bmp + +This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool ``` ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Downloader](@ref omz_tools_downloader) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) diff --git a/samples/cpp/hello_reshape_ssd/main.cpp b/samples/cpp/hello_reshape_ssd/main.cpp index 495f4bee41d..2ae7d457cc2 100644 --- a/samples/cpp/hello_reshape_ssd/main.cpp +++ b/samples/cpp/hello_reshape_ssd/main.cpp @@ -46,8 +46,8 @@ int main(int argc, char* argv[]) { printInputAndOutputsInfo(*model); // Step 3. Validate model inputs and outputs - OPENVINO_ASSERT(model->get_parameters().size() == 1, "Sample supports models with 1 input only"); - OPENVINO_ASSERT(model->get_results().size() == 1, "Sample supports models with 1 output only"); + OPENVINO_ASSERT(model->inputs().size() == 1, "Sample supports models with 1 input only"); + OPENVINO_ASSERT(model->outputs().size() == 1, "Sample supports models with 1 output only"); // SSD has an additional post-processing DetectionOutput layer that simplifies output filtering, // try to find it. diff --git a/samples/cpp/ngraph_function_creation_sample/README.md b/samples/cpp/ngraph_function_creation_sample/README.md index aef78f0c905..390d522a13b 100644 --- a/samples/cpp/ngraph_function_creation_sample/README.md +++ b/samples/cpp/ngraph_function_creation_sample/README.md @@ -1,82 +1,62 @@ # nGraph Function Creation C++ Sample {#openvino_inference_engine_samples_ngraph_function_creation_sample_README} -This sample demonstrates how to execute an synchronous inference using [OpenVINO Model feature](../../../docs/OV_Runtime_UG/model_representation.md) to create a network, which uses weights from LeNet classification network, which is known to work well on digit classification tasks. +This sample demonstrates how to execute an synchronous inference using [model](../../../docs/OV_Runtime_UG/model_representation.md) built on the fly which uses weights from LeNet classification model, which is known to work well on digit classification tasks. -The sample supports only single-channel [MNIST database](http://yann.lecun.com/exdb/mnist) images as an input. +You do not need an XML file to create a model. The API of ngraph::Function allows creating a model on the fly from the source code. -You do not need an XML file to create a network. The API of ngraph::Function allows to create a network on the fly from the source code. +The following C++ API is used in the application: -nGraph Function Creation C++ Sample demonstrates the following Inference Engine API in your applications: +| Feature | API | Description | +| :--- | :--- | :--- | +| OpenVINO Runtime Info | `ov::Core::get_versions` | Get device plugins versions | +| Shape Operations | `ov::Output::get_shape`, `ov::Shape::size`, `ov::shape_size`| Operate with shape | +| Tensor Operations | `ov::Tensor::get_byte_size`, `ov::Tensor:data` | Get tensor byte size and its data | +| Model Operations | `ov::set_batch` | Operate with model batch size | +| Infer Request Operations | `ov::InferRequest::get_input_tensor` | Get a input tensor | +| nGraph Functions | `ov::opset8::Parameter`, `ov::Node::output`, `ov::opset8::Constant`, `ov::opset8::Convolution`, `ov::opset8::Add`, `ov::opset1::MaxPool`, `ov::opset8::Reshape`, `ov::opset8::MatMul`, `ov::opset8::Relu`, `ov::opset8::Softmax`, `ov::descriptor::Tensor::set_names`, `ov::opset8::Result`, `ov::Model`, `ov::ParameterVector::vector` | Used to construct an nGraph function | -| Feature | API | Description | -|:--- |:--- |:--- -|OpenVINO Runtime Version| `ov::get_openvino_versio` | Get Openvino API version -|Available Devices|`ov::Core::get_available_devices`| Get version information of the devices for inference -| Model Operations | `ov::set_batch`, `ov::get_batch` | Managing of model, operate with its batch size. Setting batch size using input image count. -|nGraph Functions| `ov::op`, `ov::Node`, `ov::Shape::Shape`, `ov::Strides::Strides`, `ov::CoordinateDiff::CoordinateDiff`, `ov::Node::set_friendly_name`, `ov::shape_size`, `ov::ParameterVector::vector` | Illustrates how to construct an nGraph function +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). - -| Options | Values | -|:--- |:--- -| Validated Models | LeNet -| Model Format | Network weights file (\*.bin) -| Validated images | single-channel `MNIST ubyte` images -| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | -| Other language realization | [Python](../../../samples/python/ngraph_function_creation_sample/README.md) | +| Options | Values | +| :--- | :--- | +| Validated Models | LeNet | +| Model Format | model weights file (\*.bin) | +| Validated images | single-channel `MNIST ubyte` images | +| Supported devices | [All](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Other language realization | [Python](../../../samples/python/ngraph_function_creation_sample/README.md) | ## How It Works -At startup, the sample application reads command-line parameters, prepares input data, creates a network using the [OpenVINO Model feature](../../../docs/OV_Runtime_UG/model_representation.md) and passed weights file, loads the network and image(s) to the Inference Engine plugin, performs synchronous inference and processes output data, logging each step in a standard output stream. You can place labels in .labels file near the model to get pretty output. +At startup, the sample application does the following: +- Reads command line parameters +- [Build a Model](../../../docs/OV_Runtime_UG/model_representation.md) and passed weights file +- Loads the model and input data to the OpenVINO™ Runtime plugin +- Performs synchronous inference and processes output data, logging each step in a standard output stream -You can see the explicit description of each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +You can see the explicit description of each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running -To run the sample, you need specify a model wights and MNIST ubyte image: - -- you can use LeNet model weights in the sample folder: `lenet.bin` with FP32 weights file -- you can use images from the media files collection available at https://storage.openvinotoolkit.org/data/test_data. +``` +ngraph_function_creation_sample +``` > **NOTES**: > +> - you can use LeNet model weights in the sample folder: `lenet.bin` with FP32 weights file > - The `lenet.bin` with FP32 weights file was generated by the [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) tool from the public LeNet model with the `--input_shape [64,1,28,28]` parameter specified. > > The original model is available in the [Caffe* repository](https://github.com/BVLC/caffe/tree/master/examples/mnist) on GitHub\*. -Running the application with the `-h` option yields the following usage message: - -``` -ngraph_function_creation_sample -h -[ INFO ] InferenceEngine: - API version ............ - Build .................. - Description ....... API -[ INFO ] Parsing input parameters - -ngraph_function_creation_sample [OPTION] -Options: - - -h Print a usage message. - -m "" Required. Path to a .bin file with weights for the trained model. - -i "" Required. Path to a folder with images or path to image files. Support ubyte files only. - -d "" Optional. Specify the target device to infer on (the list of available devices is shown below). Default value is CPU. Use "-d HETERO:" format to specify HETERO plugin. Sample will look for a suitable plugin for device specified. - -nt "" Number of top results. The default value is 10. - -Available target devices: - -``` - -Running the application with the empty list of options yields the usage message given above and an error message. You can do inference of an image using a pre-trained model on a GPU using the following command: ``` -/ngraph_function_creation_sample -m /lenet.bin -i -d GPU +ngraph_function_creation_sample lenet.bin GPU ``` ## Sample Output @@ -84,46 +64,102 @@ You can do inference of an image using a pre-trained model on a GPU using the fo The sample application logs each step in a standard output stream and outputs top-10 inference results. ``` -ngraph_function_creation_sample.exe -m lenet.bin -i 7-ubyte -[ INFO ] InferenceEngine: - API version ............ - Build .................. - Description ....... API -[ INFO ] Parsing input parameters -[ INFO ] Files were added: 1 -[ INFO ] 7-ubyte -[ INFO ] Loading Inference Engine +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] [ INFO ] Device info: - CPU - openvino_intel_cpu_plugin version ......... - Build ........... - -[ INFO ] Preparing input blobs -[ INFO ] Batch size is 1 -[ INFO ] Checking that the outputs are as the sample expects -[ INFO ] Loading model to the device +[ INFO ] GPU +[ INFO ] Intel GPU plugin version ......... +[ INFO ] Build ........... +[ INFO ] +[ INFO ] +[ INFO ] Create model from weights: lenet.bin +[ INFO ] model name: lenet +[ INFO ] inputs +[ INFO ] input name: NONE +[ INFO ] input type: f32 +[ INFO ] input shape: {64, 1, 28, 28} +[ INFO ] outputs +[ INFO ] output name: output_tensor +[ INFO ] output type: f32 +[ INFO ] output shape: {64, 10} +[ INFO ] Batch size is 10 +[ INFO ] model name: lenet +[ INFO ] inputs +[ INFO ] input name: NONE +[ INFO ] input type: u8 +[ INFO ] input shape: {10, 28, 28, 1} +[ INFO ] outputs +[ INFO ] output name: output_tensor +[ INFO ] output type: f32 +[ INFO ] output shape: {10, 10} +[ INFO ] Compiling a model for the GPU device [ INFO ] Create infer request -[ INFO ] Start inference -[ INFO ] Processing output blobs +[ INFO ] Combine images in batch and set to input tensor +[ INFO ] Start sync inference +[ INFO ] Processing output tensor -Top 10 results: +Top 1 results: -Image 7-ubyte +Image 0 -classid probability -------- ----------- -7 1.0000000 -4 0.0000000 -8 0.0000000 -9 0.0000000 -5 0.0000000 -3 0.0000000 -1 0.0000000 -0 0.0000000 -2 0.0000000 -6 0.0000000 +classid probability label +------- ----------- ----- +0 1.0000000 0 -[ INFO ] This sample is an API example, for performance measurements, use the dedicated benchmark_app tool +Image 1 + +classid probability label +------- ----------- ----- +1 1.0000000 1 + +Image 2 + +classid probability label +------- ----------- ----- +2 1.0000000 2 + +Image 3 + +classid probability label +------- ----------- ----- +3 1.0000000 3 + +Image 4 + +classid probability label +------- ----------- ----- +4 1.0000000 4 + +Image 5 + +classid probability label +------- ----------- ----- +5 1.0000000 5 + +Image 6 + +classid probability label +------- ----------- ----- +6 1.0000000 6 + +Image 7 + +classid probability label +------- ----------- ----- +7 1.0000000 7 + +Image 8 + +classid probability label +------- ----------- ----- +8 1.0000000 8 + +Image 9 + +classid probability label +------- ----------- ----- +9 1.0000000 9 ``` @@ -146,6 +182,6 @@ classid probability ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) diff --git a/samples/cpp/speech_sample/README.md b/samples/cpp/speech_sample/README.md index 4a09a8b37e4..716b90d3b96 100644 --- a/samples/cpp/speech_sample/README.md +++ b/samples/cpp/speech_sample/README.md @@ -4,53 +4,50 @@ This sample demonstrates how to execute an Asynchronous Inference of acoustic mo The sample works with Kaldi ARK or Numpy* uncompressed NPZ files, so it does not cover an end-to-end speech recognition scenario (speech to text), requiring additional preprocessing (feature extraction) to get a feature vector from a speech signal, as well as postprocessing (decoding) to produce text from scores. -Automatic Speech Recognition C++ sample application demonstrates how to use the following Inference Engine C++ API in applications: +The following C++ API is used in the application: -| Feature | API | Description | -|:--- |:--- |:--- -|OpenVINO Runtime Version| `ov::get_openvino_versio` | Get Openvino API version -|Available Devices|`ov::Core::get_available_devices`| Get version information of the devices for inference -| Model Operations | `ov::set_batch`, `ov::get_batch` | Managing of model, operate with its batch size. Setting batch size using input image count. -|Model Operations|`ov::Model::add_output`| Change names of output layers in the model -|Import Model|`ov::CompiledModel::export_model`,`ov::Core::import_model`| Creates a CompiledModel from a previously exported model -|Asynchronous Infer| `ov::InferRequest::start_async`, `ov::InferRequest::wait`| Do asynchronous inference and waits until inference result becomes available -|InferRequest Operations|`ov::InferRequest::query_state`, `ov::VariableState::reset`| Gets and resets state control interface for given CompiledModel -|InferRequest Operations|`ov::InferRequest::get_profiling_info`| Get profiling info for infer request +| Feature | API | Description | +| :--- | :--- | :--- | +| Available Devices | `ov::Core::get_available_devices`, `ov::Core::get_property` | Get information of the devices for inference | +| Import/Export Model | `ov::Core::import_model`, `ov::CompiledModel::export_model` | The GNA plugin supports loading and saving of the GNA-optimized model | +| Model Operations | `ov::set_batch`, `ov::Model::add_output`, `ov::CompiledModel::inputs`, `ov::CompiledModel::outputs` | Managing of model: configure batch_size, input and output tensors | +| Node Operations | `ov::OutputVector::size`, `ov::Output::get_shape` | Get node shape | +| Asynchronous Infer | `ov::InferRequest::start_async`, `ov::InferRequest::wait` | Do asynchronous inference and waits until inference result becomes available | +| InferRequest Operations | `ov::InferRequest::query_state`, `ov::VariableState::reset` | Gets and resets CompiledModel state control | +| Tensor Operations | `ov::Tensor::get_size`, `ov::Tensor::data`, `ov::InferRequest::get_tensor` | Get a tensor, its size and data | +| Profiling | `ov::InferRequest::get_profiling_info` | Get infer request profiling info | -Basic Inference Engine API is covered by [Hello Classification C++ sample](../hello_classification/README.md). +Basic OpenVINO™ Runtime API is covered by [Hello Classification C++ sample](../hello_classification/README.md). -| Options | Values | -|:--- |:--- -| Validated Models | Acoustic model based on Kaldi\* neural networks (see [Model Preparation](#model-preparation) section) -| Model Format | Inference Engine Intermediate Representation (\*.xml + \*.bin) -| Supported devices | See [Execution Modes](#execution-modes) section below and [List Supported Devices](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | +| Options | Values | +| :--- | :--- | +| Validated Models | Acoustic model based on Kaldi\* neural networks (see [Model Preparation](#model-preparation) section) | +| Model Format | OpenVINO™ toolkit Intermediate Representation (\*.xml + \*.bin) | +| Supported devices | See [Execution Modes](#execution-modes) section below and [List Supported Devices](../../../docs/OV_Runtime_UG/supported_plugins/Supported_Devices.md) | ## How It Works -On startup, the application reads command line parameters, loads a specified model and input data to the Inference Engine plugin, performs synchronous inference on all speech utterances stored in the input file. Context-windowed speech frames are processed in batches of 1-8 -frames according to the `-bs` parameter. Batching across utterances is not supported by this sample. When inference is done, the application creates an output file. If the `-r` option is given, error -statistics are provided for each speech utterance as shown above. +At startup, the sample application reads command-line parameters, loads a specified model and input data to the OpenVINO™ Runtime plugin, performs inference on all speech utterances stored in the input file(s), logging each step in a standard output stream. +If the `-r` option is given, error statistics are provided for each speech utterance as shown above. You can see the explicit description of -each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the Inference Engine with Your Application" guide. +each sample step at [Integration Steps](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) section of "Integrate the OpenVINO™ Runtime with Your Application" guide. ### GNA-specific details #### Quantization -If the GNA device is selected (for example, using the `-d` GNA flag), the GNA Inference Engine plugin quantizes the model and input feature vector sequence to integer representation before performing inference. +If the GNA device is selected (for example, using the `-d` GNA flag), the GNA OpenVINO™ Runtime plugin quantizes the model and input feature vector sequence to integer representation before performing inference. Several parameters control neural network quantization. The `-q` flag determines the quantization mode. Three modes are supported: -- *static* - The first -utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used -for all subsequent inputs. The neural network is quantized to accommodate the scaled input dynamic range. -- *dynamic* - The scale factor for each input batch is computed -just before inference on that batch. The input and network are (re)quantized on the fly using an efficient procedure. +- *static* - The first utterance in the input file is scanned for dynamic range. The scale factor (floating point scalar multiplier) required to scale the maximum input value of the first utterance to 16384 (15 bits) is used for all subsequent inputs. The neural network is quantized to accommodate the scaled input dynamic range. +- *dynamic* - The scale factor for each input batch is computed just before inference on that batch. The input and network are (re)quantized on the fly using an efficient procedure. - *user-defined* - The user may specify a scale factor via the `-sf` flag that will be used for static quantization. -The `-qb` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers. For example, when `-qb 8` is specified, the plugin will use 8-bit weights wherever possible in the +The `-qb` flag provides a hint to the GNA plugin regarding the preferred target weight resolution for all layers. For example, when `-qb 8` is specified, the plugin will use 8-bit weights wherever possible in the network. + > **NOTE**: > > - It is not always possible to use 8-bit weights due to GNA hardware limitations. For example, convolutional layers always use 16-bit weights (GNA hardware version 1 and 2). This limitation will be removed in GNA hardware version 3 and higher. @@ -80,131 +77,129 @@ In addition to performing inference directly from a GNA model file, these combin ## Building -To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in Inference Engine Samples guide. +To build the sample, please use instructions available at [Build the Sample Applications](../../../docs/OV_Runtime_UG/Samples_Overview.md) section in OpenVINO™ Toolkit Samples guide. ## Running -Running the application with the `-h` option yields the following usage message: +Run the application with the -h option to see the usage message: -```sh -./speech_sample -h -[ INFO ] InferenceEngine: - API version ............ - Build .................. - Description ....... API +``` +speech_sample -h +``` + +Usage message: + +``` +[ INFO ] OpenVINO Runtime version ......... +[ INFO ] Build ........... +[ INFO ] [ INFO ] Parsing input parameters speech_sample [OPTION] Options: - -h Print a usage message. - -i "" Required. Paths to input files. Example of usage: or or . - -m "" Required. Path to an .xml file with a trained model (required if -rg is missing). - -o "" Optional. Output file name to save scores. Example of usage: or - -d "" Optional. Specify a target device to infer on. CPU, GPU, MYRIAD, GNA_AUTO, GNA_HW, GNA_SW_FP32, GNA_SW_EXACT and HETERO with combination of GNA - as the primary device and CPU as a secondary (e.g. HETERO:GNA,CPU) are supported. The list of available devices is shown below. The sample will look for a suitable plugin for device specified. - -pc Optional. Enables per-layer performance report. - -q "" Optional. Input quantization mode: static (default), dynamic, or user (use with -sf). - -qb "" Optional. Weight bits for quantization: 8 or 16 (default) - -sf "" Optional. User-specified input scale factor for quantization (use with -q user). If the network contains multiple inputs, provide scale factors by separating them with commas. - -bs "" Optional. Batch size 1-8 (default 1) - -r "" Optional. Read referefile and compare scores. Example of usage: or - -rg "" Read GNA model from file using path/filename provided (required if -m is missing). - -wg "" Optional. Write GNA model to file using path/filename provided. - -we "" Optional. Write GNA embedded model to file using path/filename provided. - -nthreads "" Optional. Number of threads to use for concurrent async inference requests on the GNA. - -cw_l "" Optional. Number of frames for left context windows (default is 0). Works only with context window networks. - If you use the cw_l or cw_r flag, then batch size and nthreads arguments are ignored. - -cw_r "" Optional. Number of frames for right context windows (default is 0). Works only with context window networks. - If you use the cw_r or cw_l flag, then batch size and nthreads arguments are ignored. - -oname "" Optional. Layer names for output blobs. The names are separated with "," Example: Output1:port,Output2:port - -iname "" Optional. Layer names for input blobs. The names are separated with "," Example: Input1,Input2 - -pwl_me "" Optional. The maximum percent of error for PWL function.The value must be in <0, 100> range. The default value is 1.0. - -Available target devices: + -h Print a usage message. + -i "" Required. Paths to input files. Example of usage: or or . + -m "" Required. Path to an .xml file with a trained model (required if -rg is missing). + -o "" Optional. Output file name to save scores. Example of usage: or + -d "" Optional. Specify a target device to infer on. CPU, GPU, MYRIAD, GNA_AUTO, GNA_HW, GNA_HW_WITH_SW_FBACK, GNA_SW_FP32, GNA_SW_EXACT and HETERO with combination of GNA as the primary device and CPU as a secondary (e.g. HETERO:GNA,CPU) are supported. The sample will look for a suitable plugin for device specified. + -pc Optional. Enables per-layer performance report. + -q "" Optional. Input quantization mode: static (default), dynamic, or user (use with -sf). + -qb "" Optional. Weight bits for quantization: 8 or 16 (default) + -sf "" Optional. User-specified input scale factor for quantization (use with -q user). If the network contains multiple inputs, provide scale factors by separating them with commas. + -bs "" Optional. Batch size 1-8 (default 1) + -r "" Optional. Read reference score file and compare scores. Example of usage: or + -rg "" Read GNA model from file using path/filename provided (required if -m is missing). + -wg "" Optional. Write GNA model to file using path/filename provided. + -we "" Optional. Write GNA embedded model to file using path/filename provided. + -cw_l "" Optional. Number of frames for left context windows (default is 0). Works only with context window networks. If you use the cw_l or cw_r flag, then batch size argument is ignored. + -cw_r "" Optional. Number of frames for right context windows (default is 0). Works only with context window networks. If you use the cw_r or cw_l flag, then batch size argument is ignored. + -oname "" Optional. Layer names for output blobs. The names are separated with "," Example: Output1:port,Output2:port + -iname "" Optional. Layer names for input blobs. The names are separated with "," Example: Input1,Input2 + -pwl_me "" Optional. The maximum percent of error for PWL function.The value must be in <0, 100> range. The default value is 1.0. + -exec_target "" Optional. Specify GNA execution target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_EXEC_TARGET config option description. + -compile_target "" Optional. Specify GNA compile target generation. May be one of GNA_TARGET_2_0, GNA_TARGET_3_0. By default, generation corresponds to the GNA HW available in the system or the latest fully supported generation by the software. See the GNA Plugin's GNA_COMPILE_TARGET config option description. +Available target devices: CPU GNA GPU ``` -Running the application with the empty list of options yields the usage message given above and an error message. - ### Model Preparation -Run Model Optimizer to convert a Kaldi nnet1 or nnet2 neural network to Inference Engine Intermediate Representation format: +You can use the following model optimizer command to convert a Kaldi nnet1 or nnet2 neural model to OpenVINO™ toolkit Intermediate Representation format: +``` mo --framework kaldi --input_model wsj_dnn5b.nnet --counts wsj_dnn5b.counts --remove_output_softmax --output_dir - -Assuming that the Kaldi-trained neural network, `wsj_dnn5b.nnet`, and Kaldi class counts file, `wsj_dnn5b.counts`, are in the working directory this produces the IR network consisting of `wsj_dnn5b.xml` and `wsj_dnn5b.bin`. +``` The following pre-trained models are available: -- wsj\_dnn5b\_smbr -- rm\_lstm4f -- rm\_cnn4a\_smbr +- wsj_dnn5b_smbr +- rm_lstm4f +- rm_cnn4a_smbr -All of them can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/). +All of them can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2). ### Speech Inference -Once the IR is created, you can use the following command to do inference on Intel® Processors with the GNA co-processor (or emulation library): +Once the IR is created, you can do inference on Intel® Processors with the GNA co-processor (or emulation library): -```sh -/speech_sample -d GNA_AUTO -bs 2 -i /dev93_10.ark -m /wsj_dnn5b.xml -o scores.ark -r /dev93_scores_10.ark +``` +speech_sample -m wsj_dnn5b.xml -i dev93_10.ark -r dev93_scores_10.ark -d GNA_AUTO -o result.ark ``` -Here, the floating point Kaldi-generated reference neural network scores (`dev93_scores_10.ark`) corresponding to the input feature file (`dev93_10.ark`) are assumed to be available -for comparison. - -All of them can be downloaded from [https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/wsj_dnn5b_smbr](https://storage.openvinotoolkit.org/models_contrib/speech/2021.2/wsj_dnn5b_smbr). Inference Engine Intermediate Representation `wsj_dnn5b.xml` file was generated in the previous [Model Preparation](#model-preparation) section. +Here, the floating point Kaldi-generated reference neural network scores (`dev93_scores_10.ark`) corresponding to the input feature file (`dev93_10.ark`) are assumed to be available for comparison. > **NOTES**: > -> - Before running the sample with a trained model, make sure the model is converted to the Inference Engine format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). - +> - Before running the sample with a trained model, make sure the model is converted to the intermediate representation (IR) format (\*.xml + \*.bin) using the [Model Optimizer tool](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md). +> +> - The sample supports input and output in numpy file format (.npz) ## Sample Output -The acoustic log likelihood sequences for all utterances are stored in the file. Example `scores.ark` or `scores.npz`. If the `-r` option is used, a report on the statistical score error is generated for each utterance such as -the following: +The sample application logs each step in a standard output stream. -```sh -./speech_sample -d GNA_AUTO -bs 2 -i dev93_10.ark -m wsj_dnn5b.xml -o scores.ark -r dev93_scores_10.ark -[ INFO ] InferenceEngine: - API version ............ - Build .................. - Description ....... API +``` +[ INFO ] OpenVINO runtime: OpenVINO Runtime version ......... 2022.1.0 +[ INFO ] Build ........... 2022.1.0-6311-a90bb1ff017 +[ INFO ] [ INFO ] Parsing input parameters -[ INFO ] No extensions provided -[ INFO ] Loading Inference Engine -[ INFO ] Device info: - GNA - GNAPlugin version ......... - Build ........... - -[ INFO ] Loading network files -[ INFO ] Batch size is 2 +[ INFO ] Loading model files: +[ INFO ] \test_data\models\wsj_dnn5b_smbr_fp32\wsj_dnn5b_smbr_fp32.xml [ INFO ] Using scale factor of 2175.43 calculated from first utterance. +[ INFO ] Model loading time 0.0034 ms +[ INFO ] Loading model to the device GNA_AUTO [ INFO ] Loading model to the device -[ INFO ] Model loading time 124.21 ms -[ INFO ] Number scores per frame : 6850 +[ INFO ] Number scores per frame : 3425 Utterance 0: -Total time in Infer (HW and SW): 2291.64 ms +Total time in Infer (HW and SW): 5687.53 ms Frames in utterance: 1294 frames -Average Infer time per frame: 1.77097 ms +Average Infer time per frame: 4.39531 ms max error: 0.705184 avg error: 0.0448388 - avg rms error: 0.0287733 + avg rms error: 0.0574098 stdev error: 0.0371649 End of Utterance 0 -[ INFO ] Number scores per frame : 6850 -Utterance X: +[ INFO ] Number scores per frame : 3425 +Utterance 1: +Total time in Infer (HW and SW): 4341.34 ms +Frames in utterance: 1005 frames +Average Infer time per frame: 4.31974 ms + max error: 0.757597 + avg error: 0.0452166 + avg rms error: 0.0578436 + stdev error: 0.0372769 + + +End of Utterance 1 + ... End of Utterance X -[ INFO ] Execution successful -[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool +[ INFO ] Execution successful ``` ## Use of Sample in Kaldi* Speech Recognition Pipeline @@ -223,7 +218,7 @@ nnet-forward --use-gpu=no final.feature_transform "ark,s,cs:copy-feats scp:feats ```sh ./speech_sample -d GNA_AUTO -bs 8 -i feat.ark -m wsj_dnn5b.xml -o scores.ark ``` -Inference Engine Intermediate Representation `wsj_dnn5b.xml` file was generated in the previous [Model Preparation](#model-preparation) section. +OpenVINO™ toolkit Intermediate Representation `wsj_dnn5b.xml` file was generated in the previous [Model Preparation](#model-preparation) section. 3. Run the Kaldi decoder to produce n-best text hypotheses and select most likely text given the WFST (`HCLG.fst`), vocabulary (`words.txt`), and TID/PID mapping (`final.mdl`): @@ -241,7 +236,7 @@ All of mentioned files can be downloaded from [https://storage.openvinotoolkit.o ## See Also -- [Integrate the Inference Engine with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) -- [Using Inference Engine Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) +- [Integrate the OpenVINO™ Runtime with Your Application](../../../docs/OV_Runtime_UG/Integrate_with_customer_application_new_API.md) +- [Using OpenVINO™ Toolkit Samples](../../../docs/OV_Runtime_UG/Samples_Overview.md) - [Model Downloader](@ref omz_tools_downloader) - [Model Optimizer](../../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md)