benchmark_app: factor out advanced options (#15605)
* benchmark_app: factor out advanced options * cpp/benchmark_app: add usage word * move api to advanced, group b/shape/data_shape/layout into Input shapes, factor out Statistics dumping options * Factor out Device-specific performance options * Factor out Preprocessing options * Minor regroup
This commit is contained in:
Zlobin Vladimir authored and 
GitHub committed
1 parent
c8afb90218
commit
ee2e9d497c
7 files changed
+523
-456
No files matched your search
+148
-103
@@ -8,7 +8,7 @@ This page demonstrates how to use the Benchmark Python Tool to estimate deep lea
|
||||
|
||||
The Python benchmark_app is automatically installed when you install OpenVINO Developer Tools using [PyPI](../../docs/install_guides/installing-openvino-pip.md). Before running `benchmark_app`, make sure the `openvino_env` virtual environment is activated, and navigate to the directory where your model is located.
|
||||
|
||||
The benchmarking application works with models in the OpenVINO IR (`model.xml` and `model.bin`) and ONNX (`model.onnx`) formats. Make sure to [convert your models](../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) if necessary.
|
||||
The benchmarking application works with models in the OpenVINO IR (`model.xml` and `model.bin`) and ONNX (`model.onnx`) formats. Make sure to [convert your models](../../docs/MO_DG/Deep_Learning_Model_Optimizer_DevGuide.md) if necessary.
|
||||
|
||||
To run benchmarking with default options on a model, use the following command:
|
||||
|
||||
@@ -82,7 +82,7 @@ The benchmark tool runs benchmarking on user-provided input images in `.jpg`, `.
|
||||
./benchmark_app -m model.xml -i test1.jpg
|
||||
```
|
||||
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the `-i` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
The tool will repeatedly loop through the provided inputs and run inferencing on them for the specified amount of time or number of iterations. If the `-i` flag is not used, the tool will automatically generate random data to fit the input shape of the model.
|
||||
|
||||
### Examples
|
||||
For more usage examples (and step-by-step instructions on how to set up a model for benchmarking), see the [Examples of Running the Tool](#examples-of-running-the-tool-python) section.
|
||||
@@ -106,126 +106,109 @@ Running the application with the `-h` or `--help` option yields the following us
|
||||
```
|
||||
[Step 1/11] Parsing and validating input arguments
|
||||
[ INFO ] Parsing input parameters
|
||||
usage: benchmark_app.py [-h [HELP]] [-i PATHS_TO_INPUT [PATHS_TO_INPUT ...]] -m PATH_TO_MODEL [-d TARGET_DEVICE] [-extensions EXTENSIONS] [-c PATH_TO_CLDNN_CONFIG]
|
||||
[-hint {throughput,cumulative_throughput,latency,none}] [-api {sync,async}] [-niter NUMBER_ITERATIONS] [-nireq NUMBER_INFER_REQUESTS] [-b BATCH_SIZE] [-t TIME]
|
||||
[-shape SHAPE] [-data_shape DATA_SHAPE] [-layout LAYOUT] [-nstreams NUMBER_STREAMS] [-latency_percentile LATENCY_PERCENTILE] [-nthreads NUMBER_THREADS]
|
||||
[-pin {YES,NO,NUMA,HYBRID_AWARE}] [-exec_graph_path EXEC_GRAPH_PATH] [-pc [PERF_COUNTS]] [-pcsort {no_sort,sort,simple_sort}] [-pcseq [PCSEQ]]
|
||||
[-inference_only [INFERENCE_ONLY]] [-report_type {no_counters,average_counters,detailed_counters}] [-report_folder REPORT_FOLDER] [-dump_config DUMP_CONFIG]
|
||||
[-load_config LOAD_CONFIG] [-infer_precision INFER_PRECISION]
|
||||
[-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}]]
|
||||
[-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}]]
|
||||
[-iop INPUT_OUTPUT_PRECISION]
|
||||
[-cdir CACHE_DIR] [-lfile [LOAD_FROM_FILE]] [--mean_values [R,G,B]] [--scale_values [R,G,B]]
|
||||
usage: benchmark_app.py [-h [HELP]] [-i PATHS_TO_INPUT [PATHS_TO_INPUT ...]] -m PATH_TO_MODEL [-d TARGET_DEVICE]
|
||||
[-hint {throughput,cumulative_throughput,latency,none}] [-niter NUMBER_ITERATIONS] [-t TIME] [-b BATCH_SIZE] [-shape SHAPE]
|
||||
[-data_shape DATA_SHAPE] [-layout LAYOUT] [-extensions EXTENSIONS] [-c PATH_TO_CLDNN_CONFIG] [-cdir CACHE_DIR] [-lfile [LOAD_FROM_FILE]]
|
||||
[-api {sync,async}] [-nireq NUMBER_INFER_REQUESTS] [-nstreams NUMBER_STREAMS] [-inference_only [INFERENCE_ONLY]]
|
||||
[-infer_precision INFER_PRECISION] [-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}]
|
||||
[-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}] [-iop INPUT_OUTPUT_PRECISION] [--mean_values [R,G,B]] [--scale_values [R,G,B]]
|
||||
[-nthreads NUMBER_THREADS] [-pin {YES,NO,NUMA,HYBRID_AWARE}] [-latency_percentile LATENCY_PERCENTILE]
|
||||
[-report_type {no_counters,average_counters,detailed_counters}] [-report_folder REPORT_FOLDER] [-pc [PERF_COUNTS]]
|
||||
[-pcsort {no_sort,sort,simple_sort}] [-pcseq [PCSEQ]] [-exec_graph_path EXEC_GRAPH_PATH] [-dump_config DUMP_CONFIG] [-load_config LOAD_CONFIG]
|
||||
|
||||
Options:
|
||||
-h [HELP], --help [HELP]
|
||||
Show this help message and exit.
|
||||
|
||||
-i PATHS_TO_INPUT [PATHS_TO_INPUT ...], --paths_to_input PATHS_TO_INPUT [PATHS_TO_INPUT ...]
|
||||
Optional. Path to a folder with images and/or binaries or to specific image or binary file.It is also allowed to map files to model inputs:
|
||||
input_1:file_1/dir1,file_2/dir2,input_4:file_4/dir4 input_2:file_3/dir3
|
||||
Currently supported data types: bin, npy. If OPENCV is enabled, this functionality
|
||||
is extended with the following data types: bmp, dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
input_1:file_1/dir1,file_2/dir2,input_4:file_4/dir4 input_2:file_3/dir3 Currently supported data types: bin, npy. If OPENCV is enabled, this
|
||||
functionalityis extended with the following data types: bmp, dib, jpeg, jpg, jpe, jp2, png, pbm, pgm, ppm, sr, ras, tiff, tif.
|
||||
|
||||
-m PATH_TO_MODEL, --path_to_model PATH_TO_MODEL
|
||||
Required. Path to an .xml/.onnx file with a trained model or to a .blob file with a trained compiled model.
|
||||
|
||||
-d TARGET_DEVICE, --target_device TARGET_DEVICE
|
||||
Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use '-d HETERO:<comma separated
|
||||
devices list>' format to specify HETERO plugin. Use '-d MULTI:<comma separated devices list>' format to specify MULTI plugin. The application looks for
|
||||
a suitable plugin for the specified device.
|
||||
-extensions EXTENSIONS, --extensions EXTENSIONS
|
||||
Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.
|
||||
-c PATH_TO_CLDNN_CONFIG, --path_to_cldnn_config PATH_TO_CLDNN_CONFIG
|
||||
Optional. Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
Optional. Specify a target device to infer on (the list of available devices is shown below). Default value is CPU. Use '-d HETERO:<comma
|
||||
separated devices list>' format to specify HETERO plugin. Use '-d MULTI:<comma separated devices list>' format to specify MULTI plugin. The
|
||||
application looks for a suitable plugin for the specified device.
|
||||
|
||||
-hint {throughput,cumulative_throughput,latency,none}, --perf_hint {throughput,cumulative_throughput,latency,none}
|
||||
Optional. Performance hint (latency or throughput or cumulative_throughput or none). Performance hint allows the OpenVINO device to select the right
|
||||
model-specific settings. 'throughput': device performance mode will be set to THROUGHPUT. 'cumulative_throughput': device performance mode will be set
|
||||
to CUMULATIVE_THROUGHPUT. 'latency': device performance mode will be set to LATENCY. 'none': no device performance mode will be set. Using explicit
|
||||
'nstreams' or other device-specific options, please set hint to 'none'
|
||||
-api {sync,async}, --api_type {sync,async}
|
||||
Optional. Enable using sync/async API. Default value is async.
|
||||
Optional. Performance hint (latency or throughput or cumulative_throughput or none). Performance hint allows the OpenVINO device to select the
|
||||
right model-specific settings. 'throughput': device performance mode will be set to THROUGHPUT. 'cumulative_throughput': device performance
|
||||
mode will be set to CUMULATIVE_THROUGHPUT. 'latency': device performance mode will be set to LATENCY. 'none': no device performance mode will
|
||||
be set. Using explicit 'nstreams' or other device-specific options, please set hint to 'none'
|
||||
|
||||
-niter NUMBER_ITERATIONS, --number_iterations NUMBER_ITERATIONS
|
||||
Optional. Number of iterations. If not specified, the number of iterations is calculated depending on a device.
|
||||
-nireq NUMBER_INFER_REQUESTS, --number_infer_requests NUMBER_INFER_REQUESTS
|
||||
Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
|
||||
-t TIME, --time TIME Optional. Time in seconds to execute topology.
|
||||
|
||||
-api {sync,async}, --api_type {sync,async}
|
||||
Optional. Enable using sync/async API. Default value is async.
|
||||
|
||||
|
||||
Input shapes:
|
||||
-b BATCH_SIZE, --batch_size BATCH_SIZE
|
||||
Optional. Batch size value. If not specified, the batch size value is determined from Intermediate Representation
|
||||
-t TIME, --time TIME Optional. Time in seconds to execute topology.
|
||||
-shape SHAPE Optional. Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size.This parameter affect model
|
||||
Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.
|
||||
|
||||
-shape SHAPE Optional. Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. This parameter
|
||||
affect model Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.
|
||||
|
||||
-data_shape DATA_SHAPE
|
||||
Optional. Optional if model shapes are all static (original ones or set by -shape).Required if at least one input shape is dynamic and input images are
|
||||
not provided.Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or "[1,3,224,224][1,3,448,448] in case of one
|
||||
input size.
|
||||
-layout LAYOUT Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.
|
||||
-nstreams NUMBER_STREAMS, --number_streams NUMBER_STREAMS
|
||||
Optional. Number of streams to use for inference on the CPU/GPU (for HETERO and MULTI device cases use format <device1>:<nstreams1>,<device2>:<nstreams2> or just
|
||||
<nstreams>). Default value is determined automatically for a device. Please note that although the automatic selection usually provides a reasonable performance, it
|
||||
still may be non - optimal for some cases, especially for very small models. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency
|
||||
estimations the number of streams should be set to 1. See samples README for more details.
|
||||
-latency_percentile LATENCY_PERCENTILE, --latency_percentile LATENCY_PERCENTILE
|
||||
Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
-nthreads NUMBER_THREADS, --number_threads NUMBER_THREADS
|
||||
Number of threads to use for inference on the CPU, GNA (including HETERO and MULTI cases).
|
||||
-pin {YES,NO,NUMA,HYBRID_AWARE}, --infer_threads_pinning {YES,NO,NUMA,HYBRID_AWARE}
|
||||
Optional. Enable threads->cores ('YES' which is OpenVINO runtime's default for conventional CPUs), threads->(NUMA)nodes ('NUMA'), threads->appropriate
|
||||
core types ('HYBRID_AWARE', which is OpenVINO runtime's default for Hybrid CPUs) or completely disable ('NO') CPU threads pinning for CPU-involved
|
||||
inference.
|
||||
-exec_graph_path EXEC_GRAPH_PATH, --exec_graph_path EXEC_GRAPH_PATH
|
||||
Optional. Path to a file where to store executable graph information serialized.
|
||||
-pc [PERF_COUNTS], --perf_counts [PERF_COUNTS]
|
||||
Optional. Report performance counters.
|
||||
-pcsort {no_sort,sort,simple_sort}, --perf_counts_sort {no_sort,sort,simple_sort}
|
||||
Optional. Report performance counters and analysis the sort hotpoint opts. sort: Analysis opts time cost, print by hotpoint order no_sort: Analysis opts
|
||||
time cost, print by normal order simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
-pcseq [PCSEQ], --pcseq [PCSEQ]
|
||||
Optional. Report latencies for each shape in -data_shape sequence.
|
||||
-inference_only [INFERENCE_ONLY], --inference_only [INFERENCE_ONLY]
|
||||
Optional. If true inputs filling only once before measurements (default for static models), else inputs filling is included into loop measurement
|
||||
(default for dynamic models)
|
||||
-report_type {no_counters,average_counters,detailed_counters}, --report_type {no_counters,average_counters,detailed_counters}
|
||||
Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency.
|
||||
"average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model.
|
||||
"detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed infer
|
||||
request.
|
||||
-report_folder REPORT_FOLDER, --report_folder REPORT_FOLDER
|
||||
Optional. Path to a folder where statistics report is stored.
|
||||
-dump_config DUMP_CONFIG
|
||||
Optional. Path to JSON file to dump OpenVINO parameters, which were set by application.
|
||||
-load_config LOAD_CONFIG
|
||||
Optional. Path to JSON file to load custom OpenVINO parameters. Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
-infer_precision INFER_PRECISION
|
||||
Optional. Hint to specifies inference precision. Example: -infer_precision CPU:bf16,GPU:f32
|
||||
-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --input_precision {bool,f16,f32, f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all input layers of the network.
|
||||
-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --output_precision {bool,f16,f32, f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all output layers of the network.
|
||||
-iop INPUT_OUTPUT_PRECISION, --input_output_precision INPUT_OUTPUT_PRECISION
|
||||
Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required. Overwrites
|
||||
precision from ip and op options for specified layers.
|
||||
Optional. Optional if model shapes are all static (original ones or set by -shape).Required if at least one input shape is dynamic and input
|
||||
images are not provided.Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or
|
||||
"[1,3,224,224][1,3,448,448] in case of one input size.
|
||||
|
||||
-layout LAYOUT Optional. Prompts how model layouts should be treated by application. For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input
|
||||
size.
|
||||
|
||||
|
||||
Advanced options:
|
||||
-extensions EXTENSIONS, --extensions EXTENSIONS
|
||||
Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.
|
||||
|
||||
-c PATH_TO_CLDNN_CONFIG, --path_to_cldnn_config PATH_TO_CLDNN_CONFIG
|
||||
Optional. Required for GPU custom kernels. Absolute path to an .xml file with the kernels description.
|
||||
|
||||
-cdir CACHE_DIR, --cache_dir CACHE_DIR
|
||||
Optional. Enable model caching to specified directory
|
||||
|
||||
-lfile [LOAD_FROM_FILE], --load_from_file [LOAD_FROM_FILE]
|
||||
Optional. Loads model from file directly without read_model.
|
||||
|
||||
-nireq NUMBER_INFER_REQUESTS, --number_infer_requests NUMBER_INFER_REQUESTS
|
||||
Optional. Number of infer requests. Default value is determined automatically for device.
|
||||
|
||||
-nstreams NUMBER_STREAMS, --number_streams NUMBER_STREAMS
|
||||
Optional. Number of streams to use for inference on the CPU/GPU (for HETERO and MULTI device cases use format
|
||||
<device1>:<nstreams1>,<device2>:<nstreams2> or just <nstreams>). Default value is determined automatically for a device. Please note that
|
||||
although the automatic selection usually provides a reasonable performance, it still may be non - optimal for some cases, especially for very
|
||||
small models. Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency estimations the number of streams
|
||||
should be set to 1. See samples README for more details.
|
||||
|
||||
-inference_only [INFERENCE_ONLY], --inference_only [INFERENCE_ONLY]
|
||||
Optional. If true inputs filling only once before measurements (default for static models), else inputs filling is included into loop
|
||||
measurement (default for dynamic models)
|
||||
|
||||
-infer_precision INFER_PRECISION
|
||||
Optional. Specifies the inference precision. Example #1: '-infer_precision bf16'. Example #2: '-infer_precision CPU:bf16,GPU:f32'
|
||||
|
||||
-exec_graph_path EXEC_GRAPH_PATH, --exec_graph_path EXEC_GRAPH_PATH
|
||||
Optional. Path to a file where to store executable graph information serialized.
|
||||
|
||||
|
||||
Preprocessing options:
|
||||
-ip {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --input_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all input layers of the model.
|
||||
|
||||
-op {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}, --output_precision {bool,f16,f32,f64,i8,i16,i32,i64,u8,u16,u32,u64}
|
||||
Optional. Specifies precision for all output layers of the model.
|
||||
|
||||
-iop INPUT_OUTPUT_PRECISION, --input_output_precision INPUT_OUTPUT_PRECISION
|
||||
Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required.
|
||||
Overwrites precision from ip and op options for specified layers.
|
||||
|
||||
--mean_values [R,G,B]
|
||||
Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for desired input
|
||||
of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
@@ -236,6 +219,68 @@ Options:
|
||||
of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of channels depend on how the
|
||||
original model was trained. If both --mean_values and --scale_values are specified, the mean is subtracted first and then scale is applied
|
||||
regardless of the order of options in command line. Applying the values affects performance and may cause type conversion
|
||||
|
||||
|
||||
Device-specific performance options:
|
||||
-nthreads NUMBER_THREADS, --number_threads NUMBER_THREADS
|
||||
Number of threads to use for inference on the CPU, GNA (including HETERO and MULTI cases).
|
||||
|
||||
-pin {YES,NO,NUMA,HYBRID_AWARE}, --infer_threads_pinning {YES,NO,NUMA,HYBRID_AWARE}
|
||||
Optional. Enable threads->cores ('YES' which is OpenVINO runtime's default for conventional CPUs), threads->(NUMA)nodes ('NUMA'),
|
||||
threads->appropriate core types ('HYBRID_AWARE', which is OpenVINO runtime's default for Hybrid CPUs) or completely disable ('NO') CPU threads
|
||||
pinning for CPU-involved inference.
|
||||
|
||||
|
||||
Statistics dumping options:
|
||||
-latency_percentile LATENCY_PERCENTILE, --latency_percentile LATENCY_PERCENTILE
|
||||
Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).
|
||||
|
||||
-report_type {no_counters,average_counters,detailed_counters}, --report_type {no_counters,average_counters,detailed_counters}
|
||||
Optional. Enable collecting statistics report. "no_counters" report contains configuration options specified, resulting FPS and latency.
|
||||
"average_counters" report extends "no_counters" report and additionally includes average PM counters values for each layer from the model.
|
||||
"detailed_counters" report extends "average_counters" report and additionally includes per-layer PM counters and latency for each executed
|
||||
infer request.
|
||||
|
||||
-report_folder REPORT_FOLDER, --report_folder REPORT_FOLDER
|
||||
Optional. Path to a folder where statistics report is stored.
|
||||
|
||||
-pc [PERF_COUNTS], --perf_counts [PERF_COUNTS]
|
||||
Optional. Report performance counters.
|
||||
|
||||
-pcsort {no_sort,sort,simple_sort}, --perf_counts_sort {no_sort,sort,simple_sort}
|
||||
Optional. Report performance counters and analysis the sort hotpoint opts. sort: Analysis opts time cost, print by hotpoint order no_sort:
|
||||
Analysis opts time cost, print by normal order simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order
|
||||
|
||||
-pcseq [PCSEQ], --pcseq [PCSEQ]
|
||||
Optional. Report latencies for each shape in -data_shape sequence.
|
||||
|
||||
-dump_config DUMP_CONFIG
|
||||
Optional. Path to JSON file to dump OpenVINO parameters, which were set by application.
|
||||
|
||||
-load_config LOAD_CONFIG
|
||||
Optional. Path to JSON file to load custom OpenVINO parameters.
|
||||
Please note, command line parameters have higher priority then parameters from configuration file.
|
||||
Example 1: a simple JSON file for HW device with primary properties.
|
||||
{
|
||||
"CPU": {"NUM_STREAMS": "3", "PERF_COUNT": "NO"}
|
||||
}
|
||||
Example 2: a simple JSON file for meta device(AUTO/MULTI) with HW device properties.
|
||||
{
|
||||
"AUTO": {
|
||||
"PERFORMANCE_HINT": "",
|
||||
"PERF_COUNT": "NO",
|
||||
"DEVICE_PROPERTIES": {
|
||||
"CPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "3"
|
||||
},
|
||||
"GPU": {
|
||||
"INFERENCE_PRECISION_HINT": "f32",
|
||||
"NUM_STREAMS": "5"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Running the application with the empty list of options yields the usage message given above and an error message.
|
||||
|
||||
@@ -31,13 +31,13 @@ class print_help(argparse.Action):
|
||||
show_available_devices()
|
||||
sys.exit()
|
||||
class HelpFormatterWithLines(argparse.HelpFormatter):
|
||||
def _split_lines(self, text, width):
|
||||
lines = super()._split_lines(text, width)
|
||||
lines += ['']
|
||||
if "simple JSON file" not in text:
|
||||
return lines
|
||||
lines = text.split('\n')
|
||||
return lines
|
||||
def _split_lines(self, text, width):
|
||||
lines = super()._split_lines(text, width)
|
||||
lines += ['']
|
||||
if "simple JSON file" not in text:
|
||||
return lines
|
||||
lines = text.split('\n')
|
||||
return lines
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(add_help=False, formatter_class=HelpFormatterWithLines)
|
||||
@@ -60,11 +60,6 @@ def parse_args():
|
||||
'Default value is CPU. Use \'-d HETERO:<comma separated devices list>\' format to specify HETERO plugin. '
|
||||
'Use \'-d MULTI:<comma separated devices list>\' format to specify MULTI plugin. '
|
||||
'The application looks for a suitable plugin for the specified device.')
|
||||
args.add_argument('-extensions', '--extensions', type=str, required=False, default=None,
|
||||
help='Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.')
|
||||
args.add_argument('-c', '--path_to_cldnn_config', type=str, required=False,
|
||||
help='Optional. Required for GPU custom kernels. Absolute path to an .xml file with the '
|
||||
'kernels description.')
|
||||
args.add_argument('-hint', '--perf_hint', type=str, required=False, default='', choices=['throughput', 'cumulative_throughput', 'latency', 'none'],
|
||||
help='Optional. Performance hint (latency or throughput or cumulative_throughput or none). Performance hint allows the OpenVINO device to select the right model-specific settings.\n'
|
||||
'\'throughput\': device performance mode will be set to THROUGHPUT. \n'
|
||||
@@ -72,33 +67,46 @@ def parse_args():
|
||||
'\'latency\': device performance mode will be set to LATENCY. \n'
|
||||
'\'none\': no device performance mode will be set. \n'
|
||||
'Using explicit \'nstreams\' or other device-specific options, please set hint to \'none\'')
|
||||
args.add_argument('-api', '--api_type', type=str, required=False, default='async', choices=['sync', 'async'],
|
||||
help='Optional. Enable using sync/async API. Default value is async.')
|
||||
args.add_argument('-niter', '--number_iterations', type=check_positive, required=False, default=None,
|
||||
help='Optional. Number of iterations. '
|
||||
'If not specified, the number of iterations is calculated depending on a device.')
|
||||
args.add_argument('-nireq', '--number_infer_requests', type=check_positive, required=False, default=0,
|
||||
help='Optional. Number of infer requests. Default value is determined automatically for device.')
|
||||
args.add_argument('-b', '--batch_size', type=str, required=False, default='',
|
||||
help='Optional. ' +
|
||||
'Batch size value. ' +
|
||||
'If not specified, the batch size value is determined from Intermediate Representation')
|
||||
args.add_argument('-t', '--time', type=int, required=False, default=None,
|
||||
args.add_argument('-t', '--time', type=check_positive, required=False, default=None,
|
||||
help='Optional. Time in seconds to execute topology.')
|
||||
args.add_argument('-shape', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size.'
|
||||
'This parameter affect model Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.')
|
||||
args.add_argument('-data_shape', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Optional if model shapes are all static (original ones or set by -shape).'
|
||||
'Required if at least one input shape is dynamic and input images are not provided.'
|
||||
'Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or "[1,3,224,224][1,3,448,448] in case of one input size.')
|
||||
args.add_argument('-layout', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Prompts how model layouts should be treated by application. '
|
||||
'For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.')
|
||||
args.add_argument('-nstreams', '--number_streams', type=str, required=False, default=None,
|
||||
|
||||
shapes = parser.add_argument_group('Input shapes')
|
||||
shapes.add_argument('-b', '--batch_size', type=str, required=False, default='',
|
||||
help='Optional. ' +
|
||||
'Batch size value. ' +
|
||||
'If not specified, the batch size value is determined from Intermediate Representation')
|
||||
shapes.add_argument('-shape', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Set shape for input. For example, "input1[1,3,224,224],input2[1,4]" or "[1,3,224,224]" in case of one input size. '
|
||||
'This parameter affect model Parameter shape, can be dynamic. For dynamic dimesions use symbol `?`, `-1` or range `low.. up`.')
|
||||
shapes.add_argument('-data_shape', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Optional if model shapes are all static (original ones or set by -shape).'
|
||||
'Required if at least one input shape is dynamic and input images are not provided.'
|
||||
'Set shape for input tensors. For example, "input1[1,3,224,224][1,3,448,448],input2[1,4][1,8]" or "[1,3,224,224][1,3,448,448] in case of one input size.')
|
||||
shapes.add_argument('-layout', type=str, required=False, default='',
|
||||
help='Optional. '
|
||||
'Prompts how model layouts should be treated by application. '
|
||||
'For example, "input1[NCHW],input2[NC]" or "[NCHW]" in case of one input size.')
|
||||
|
||||
advs = parser.add_argument_group('Advanced options')
|
||||
advs.add_argument('-extensions', '--extensions', type=str, required=False, default=None,
|
||||
help='Optional. Path or a comma-separated list of paths to libraries (.so or .dll) with extensions.')
|
||||
advs.add_argument('-c', '--path_to_cldnn_config', type=str, required=False,
|
||||
help='Optional. Required for GPU custom kernels. Absolute path to an .xml file with the '
|
||||
'kernels description.')
|
||||
advs.add_argument('-cdir', '--cache_dir', type=str, required=False, default='',
|
||||
help="Optional. Enable model caching to specified directory")
|
||||
advs.add_argument('-lfile', '--load_from_file', required=False, nargs='?', default=argparse.SUPPRESS,
|
||||
help="Optional. Loads model from file directly without read_model.")
|
||||
args.add_argument('-api', '--api_type', type=str, required=False, default='async', choices=['sync', 'async'],
|
||||
help='Optional. Enable using sync/async API. Default value is async.')
|
||||
advs.add_argument('-nireq', '--number_infer_requests', type=check_positive, required=False, default=0,
|
||||
help='Optional. Number of infer requests. Default value is determined automatically for device.')
|
||||
advs.add_argument('-nstreams', '--number_streams', type=str, required=False, default=None,
|
||||
help='Optional. Number of streams to use for inference on the CPU/GPU '
|
||||
'(for HETERO and MULTI device cases use format <device1>:<nstreams1>,<device2>:<nstreams2> '
|
||||
'or just <nstreams>). '
|
||||
@@ -107,33 +115,45 @@ def parse_args():
|
||||
'Also, using nstreams>1 is inherently throughput-oriented option, while for the best-latency '
|
||||
'estimations the number of streams should be set to 1. '
|
||||
'See samples README for more details.')
|
||||
args.add_argument('-latency_percentile', '--latency_percentile', type=int, required=False, default=50,
|
||||
help='Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).')
|
||||
args.add_argument('-nthreads', '--number_threads', type=int, required=False, default=None,
|
||||
advs.add_argument('-inference_only', '--inference_only', type=str2bool, required=False, default=None, nargs='?', const=True,
|
||||
help='Optional. If true inputs filling only once before measurements (default for static models), '
|
||||
'else inputs filling is included into loop measurement (default for dynamic models)', )
|
||||
advs.add_argument('-infer_precision', type=str, required=False,
|
||||
help='Optional. Specifies the inference precision. Example #1: \'-infer_precision bf16\'. Example #2: \'-infer_precision CPU:bf16,GPU:f32\'')
|
||||
|
||||
prpr = parser.add_argument_group('Preprocessing options')
|
||||
prpr.add_argument('-ip', '--input_precision', type=str, required=False, choices=INPUT_OUTPUT_PRECISION_CHOICES,
|
||||
help='Optional. Specifies precision for all input layers of the model.')
|
||||
prpr.add_argument('-op', '--output_precision', type=str, required=False, choices=INPUT_OUTPUT_PRECISION_CHOICES,
|
||||
help='Optional. Specifies precision for all output layers of the model.')
|
||||
prpr.add_argument('-iop', '--input_output_precision', type=str, required=False,
|
||||
help='Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required. Overwrites precision from ip and op options for specified layers.')
|
||||
prpr.add_argument('--mean_values', type=str, required=False, default='', metavar='[R,G,B]',
|
||||
help='Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for '
|
||||
'desired input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of '
|
||||
'channels depend on how the original model was trained. Applying the values affects performance and may cause type conversion')
|
||||
prpr.add_argument('--scale_values', type=str, required=False, default='', metavar='[R,G,B]',
|
||||
help='Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for '
|
||||
'desired input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of '
|
||||
'channels depend on how the original model was trained. If both --mean_values and --scale_values are specified, the mean is '
|
||||
'subtracted first and then scale is applied regardless of the order of options in command line. Applying the values affects '
|
||||
'performance and may cause type conversion')
|
||||
|
||||
devp = parser.add_argument_group('Device-specific performance options')
|
||||
devp.add_argument('-nthreads', '--number_threads', type=int, required=False, default=None,
|
||||
help='Number of threads to use for inference on the CPU, GNA '
|
||||
'(including HETERO and MULTI cases).')
|
||||
args.add_argument('-pin', '--infer_threads_pinning', type=str, required=False, choices=['YES', 'NO', 'NUMA', 'HYBRID_AWARE'],
|
||||
devp.add_argument('-pin', '--infer_threads_pinning', type=str, required=False, choices=['YES', 'NO', 'NUMA', 'HYBRID_AWARE'],
|
||||
help='Optional. Enable threads->cores (\'YES\' which is OpenVINO runtime\'s default for conventional CPUs), '
|
||||
'threads->(NUMA)nodes (\'NUMA\'), '
|
||||
'threads->appropriate core types (\'HYBRID_AWARE\', which is OpenVINO runtime\'s default for Hybrid CPUs) '
|
||||
'or completely disable (\'NO\') '
|
||||
'CPU threads pinning for CPU-involved inference.')
|
||||
args.add_argument('-exec_graph_path', '--exec_graph_path', type=str, required=False,
|
||||
help='Optional. Path to a file where to store executable graph information serialized.')
|
||||
args.add_argument('-pc', '--perf_counts', type=str2bool, required=False, default=False, nargs='?', const=True,
|
||||
help='Optional. Report performance counters.', )
|
||||
args.add_argument('-pcsort', '--perf_counts_sort', type=str, required=False, default="",
|
||||
choices=['no_sort', 'sort', 'simple_sort'],
|
||||
help='Optional. Report performance counters and analysis the sort hotpoint opts.'
|
||||
' sort: Analysis opts time cost, print by hotpoint order'
|
||||
' no_sort: Analysis opts time cost, print by normal order'
|
||||
' simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order', )
|
||||
args.add_argument('-pcseq', '--pcseq', type=str2bool, required=False, default=False, nargs='?', const=True,
|
||||
help='Optional. Report latencies for each shape in -data_shape sequence.', )
|
||||
args.add_argument('-inference_only', '--inference_only', type=str2bool, required=False, default=None, nargs='?', const=True,
|
||||
help='Optional. If true inputs filling only once before measurements (default for static models), '
|
||||
'else inputs filling is included into loop measurement (default for dynamic models)', )
|
||||
args.add_argument('-report_type', '--report_type', type=str, required=False,
|
||||
|
||||
stat = parser.add_argument_group('Statistics dumping options')
|
||||
stat.add_argument('-latency_percentile', '--latency_percentile', type=int, required=False, default=50,
|
||||
help='Optional. Defines the percentile to be reported in latency metric. The valid range is [1, 100]. The default value is 50 (median).')
|
||||
stat.add_argument('-report_type', '--report_type', type=str, required=False,
|
||||
choices=['no_counters', 'average_counters', 'detailed_counters'],
|
||||
help="Optional. Enable collecting statistics report. \"no_counters\" report contains "
|
||||
"configuration options specified, resulting FPS and latency. \"average_counters\" "
|
||||
@@ -141,11 +161,23 @@ def parse_args():
|
||||
"counters values for each layer from the model. \"detailed_counters\" report "
|
||||
"extends \"average_counters\" report and additionally includes per-layer PM "
|
||||
"counters and latency for each executed infer request.")
|
||||
args.add_argument('-report_folder', '--report_folder', type=str, required=False, default='',
|
||||
stat.add_argument('-report_folder', '--report_folder', type=str, required=False, default='',
|
||||
help="Optional. Path to a folder where statistics report is stored.")
|
||||
args.add_argument('-dump_config', type=str, required=False, default='',
|
||||
stat.add_argument('-pc', '--perf_counts', type=str2bool, required=False, default=False, nargs='?', const=True,
|
||||
help='Optional. Report performance counters.', )
|
||||
stat.add_argument('-pcsort', '--perf_counts_sort', type=str, required=False, default="",
|
||||
choices=['no_sort', 'sort', 'simple_sort'],
|
||||
help='Optional. Report performance counters and analysis the sort hotpoint opts.'
|
||||
' sort: Analysis opts time cost, print by hotpoint order'
|
||||
' no_sort: Analysis opts time cost, print by normal order'
|
||||
' simple_sort: Analysis opts time cost, only print EXECUTED opts by normal order', )
|
||||
stat.add_argument('-pcseq', '--pcseq', type=str2bool, required=False, default=False, nargs='?', const=True,
|
||||
help='Optional. Report latencies for each shape in -data_shape sequence.', )
|
||||
advs.add_argument('-exec_graph_path', '--exec_graph_path', type=str, required=False,
|
||||
help='Optional. Path to a file where to store executable graph information serialized.')
|
||||
stat.add_argument('-dump_config', type=str, required=False, default='',
|
||||
help="Optional. Path to JSON file to dump OpenVINO parameters, which were set by application.")
|
||||
args.add_argument('-load_config', type=str, required=False, default='',
|
||||
stat.add_argument('-load_config', type=str, required=False, default='',
|
||||
help="Optional. Path to JSON file to load custom OpenVINO parameters.\n"
|
||||
"Please note, command line parameters have higher priority then parameters from configuration file.\n"
|
||||
"Example 1: a simple JSON file for HW device with primary properties.\n"
|
||||
@@ -168,28 +200,7 @@ def parse_args():
|
||||
" }\n"
|
||||
" }\n"
|
||||
" }\n"
|
||||
" }\n")
|
||||
args.add_argument('-infer_precision', type=str, required=False,
|
||||
help='Optional. Specifies the inference precision. Example #1: \'-infer_precision bf16\'. Example #2: \'-infer_precision CPU:bf16,GPU:f32\'')
|
||||
args.add_argument('-ip', '--input_precision', type=str, required=False, choices=INPUT_OUTPUT_PRECISION_CHOICES,
|
||||
help='Optional. Specifies precision for all input layers of the model.')
|
||||
args.add_argument('-op', '--output_precision', type=str, required=False, choices=INPUT_OUTPUT_PRECISION_CHOICES,
|
||||
help='Optional. Specifies precision for all output layers of the model.')
|
||||
args.add_argument('-iop', '--input_output_precision', type=str, required=False,
|
||||
help='Optional. Specifies precision for input and output layers by name. Example: -iop "input:f16, output:f16". Notice that quotes are required. Overwrites precision from ip and op options for specified layers.')
|
||||
args.add_argument('-cdir', '--cache_dir', type=str, required=False, default='',
|
||||
help="Optional. Enable model caching to specified directory")
|
||||
args.add_argument('-lfile', '--load_from_file', required=False, nargs='?', default=argparse.SUPPRESS,
|
||||
help="Optional. Loads model from file directly without read_model.")
|
||||
args.add_argument('--mean_values', type=str, required=False, default='', metavar='[R,G,B]',
|
||||
help='Optional. Mean values to be used for the input image per channel. Values to be provided in the [R,G,B] format. Can be defined for '
|
||||
'desired input of the model, for example: "--mean_values data[255,255,255],info[255,255,255]". The exact meaning and order of '
|
||||
'channels depend on how the original model was trained')
|
||||
args.add_argument('--scale_values', type=str, required=False, default='', metavar='[R,G,B]',
|
||||
help='Optional. Scale values to be used for the input image per channel. Values are provided in the [R,G,B] format. Can be defined for '
|
||||
'desired input of the model, for example: "--scale_values data[255,255,255],info[255,255,255]". The exact meaning and order of '
|
||||
'channels depend on how the original model was trained. If both --mean_values and --scale_values are specified, the mean is '
|
||||
'subtracted first and then scale is applied regardless of the order of options in command line')
|
||||
" }")
|
||||
parsed_args = parser.parse_args()
|
||||
|
||||
return parsed_args, parser
|
||||
@@ -144,7 +144,7 @@ def get_input_data(paths_to_input, app_input_info):
|
||||
return DataQueue(data, get_group_batch_sizes(app_input_info))
|
||||
|
||||
|
||||
def get_image_tensors(image_paths: List[str], info: AppInputInfo, batch_sizes: List[int]) -> List[Tensor]:
|
||||
def get_image_tensors(image_paths: List[str], info: AppInputInfo, batch_sizes: List[int]) -> List[Tensor]:
|
||||
if 'cv2' not in sys.modules:
|
||||
logger.error("Loading images requires the opencv-python or opencv-python-headless package. "
|
||||
"Please install it before continuing or run benchmark without "
|
||||
|
||||
Reference in new issue
Block a user