IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Removed cross-check-tool (#15778)

Browse Source
This commit is contained in:
Ilya Lavrenov 2023-02-20 11:09:08 +04:00 committed by GitHub
parent 699a1d1708
commit 758ebe5242
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 0 additions and 1238 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/tuning_utilities.md
View File

@ -8,6 +8,5 @@
omz_tools_accuracy_checker
omz_data_datasets
openvino_inference_engine_tools_cross_check_tool_README
@endsphinxdirective

285
tools/cross_check_tool/README.md
View File

@ -1,285 +0,0 @@
# Cross Check Tool {#openvino_inference_engine_tools_cross_check_tool_README}
Cross Check Tool is a console application that enables comparing accuracy and performance metrics for two successive
model inferences that are performed on two different supported Intel® devices or with different precisions.
The Cross Check Tool can compare the metrics per layer or all over the model.
## Running the Cross Check Tool
Cross Check Tool is distributed as a Python module and there is no need to build it. To run the Cross Check Tool,
execute the `cross_check_tool.py` file with necessary parameters. Please note that the Inference Engine assumes that weights
are in the same folder as the `.xml` file.
You can get the list of all available options using the `-h` option:
```sh
$python3 cross_check_tool.py -h
Cross Check Tool is a console application that enables comparing accuracy and
provides performance metrics
optional arguments:
-h, --help show this help message and exit
Model specific arguments:
--input INPUT, -i INPUT
Path to an input image file or multi-input file to
infer. Generates input(s) from normal distribution if
empty
--model MODEL, -m MODEL
Path to an .xml file that represents the first IR of
the trained model to infer.
--reference_model REFERENCE_MODEL, -ref_m REFERENCE_MODEL
Path to an .xml file that represents the second IR in
different precision to compare the metrics.
--layers LAYERS, -layers LAYERS
Defines layers to check. Options: all, None - for
output layers check, list of comma-separated layer
names to check. Default value is None.
--ref_layers REFERENCE_LAYERS, -reference_layers REFERENCE_LAYERS
Defines layers to check in reference model. Options: all, None - for
output layers check, list of comma-separated layer
names to check. If not specified the same layers will
be processed as in --layers parameter.
--num_of_iterations NUM_OF_ITERATIONS, -ni NUM_OF_ITERATIONS
Number of iterations to collect all over the net
performance
Plugin specific arguments:
--plugin_path PLUGIN_PATH, -pp PLUGIN_PATH
Path to a plugin folder.
--device DEVICE, -d DEVICE
The first target device to infer the model specified
with the -m or --model option. CPU, GPU or GNA are acceptable.
--config CONFIG, -conf CONFIG
Path to config file for -d or -device device plugin
--reference_device REFERENCE_DEVICE, -ref_d REFERENCE_DEVICE
The second target device to infer the model and
compare the metrics. CPU, GPU or GNA are
acceptable.
--reference_config REFERENCE_CONFIG, -ref_conf REFERENCE_CONFIG
Path to config file for -ref_d or -reference_device
device plugin
-l L Required for (CPU)-targeted custom layers.
Comma separated paths to a shared libraries with the
kernels implementation.
CCT mode arguments:
--dump Enables blobs statistics dumping
--load LOAD Path to a file to load blobs from
```
Cross Check Tool can also be installed via:
```sh
$python3 -m pip install <openvino_repo>/tools/cross_check_tool
```
In this case, to run the tool, call `cross_check_tool` on the command line with necessary parameters.
### Examples
1. To check per-layer accuracy and performance of inference in FP32 precision on the CPU against the GPU, run:
```sh
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
-m <path_to_FP32_xml> \
-d GPU \
-ref_d CPU \
--layers all
```
The output looks as follows:
```sh
[ INFO ] Cross check with one IR was enabled
[ INFO ] GPU:FP32 vs CPU:FP32
[ INFO ] The same IR on both devices: <path_to_IR>
[ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name>
[ INFO ] Layer <layer_1_name> statistics
Max absolute difference : 1.15204E-03
Min absolute difference : 0.0
Max relative difference : 1.15204E+17
Min relative difference : 0.0
Min reference value : -1.69513E+03
Min absolute reference value : 2.71080E-06
Max reference value : 1.17132E+03
Max absolute reference value : 1.69513E+03
Min actual value : -1.69513E+03
Min absolute actual value : 8.66465E-05
Max actual value : 1.17132E+03
Max absolute actual value : 1.69513E+03
Device: -d GPU -ref_d CPU
Status: OPTIMIZED_OUT OPTIMIZED_OUT
Layer type: Convolution Convolution
Real time, microsec: 0 120
Number of NAN: 0 0
Number of INF: 0 0
Number of ZERO: 0 0
...
<list_of_layer_statistics>
...
[ INFO ] Overall max absolute difference = 0.00115203857421875
[ INFO ] Overall min absolute difference = 0.0
[ INFO ] Overall max relative difference = 1.1520386483093504e+17
[ INFO ] Overall min relative difference = 0.0
[ INFO ] Execution successful
```
2. To check the overall accuracy and performance of inference on the CPU in FP32 precision against the
Intel&reg; Movidius&trade; Myriad&trade; device in FP16 precision, run:
```sh
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
-m <path_to_FP16_xml> \
-d MYRIAD \
-ref_m <path_to_FP32_xml> \
-ref_d CPU
```
The output looks as follows:
```sh
[ INFO ] Cross check with two IRs was enabled
[ INFO ] GPU:FP16 vs CPU:FP32
[ INFO ] IR for MYRIAD : <path_to_FP16_xml>
[ INFO ] IR for CPU : <path_to_FP32_xml>
[ INFO ] Statistics will be dumped for 1 layer: <output_layer_name(s)>
[ INFO ] Layer <output_layer_name> statistics
Max absolute difference : 2.32944E-02
Min absolute difference : 3.63002E-13
Max relative difference : 6.41717E+10
Min relative difference : 1.0
Min reference value : 3.63002E-13
Min absolute reference value : 3.63002E-13
Max reference value : 7.38138E-01
Max absolute reference value : 7.38138E-01
Min actual value : 0.0
Min absolute actual value : 0.0
Max actual value : 7.14844E-01
Max absolute actual value : 7.14844E-01
Device: -d MYRIAD -ref_d CPU
Status: OPTIMIZED_OUT OPTIMIZED_OUT
Layer type: Reshape Reshape
Real time, microsec: 0 0
Number of NAN: 0 0
Number of INF: 0 0
Number of ZERO: 0 0
----------------------------------------------------------------------
Overall performance, microseconds: 2.79943E+05 6.24670E+04
----------------------------------------------------------------------
[ INFO ] Overall max absolute difference = 0.023294448852539062
[ INFO ] Overall min absolute difference = 3.630019191052519e-13
[ INFO ] Overall max relative difference = 64171696128.0
[ INFO ] Overall min relative difference = 1.0
[ INFO ] Execution successful
```
3. To dump layer statistics from a specific list of layers, run:
```sh
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
-m <path_to_FP16_xml> \
-d GNA \
--dump \
--layers <comma_separated_list_of_layers>
```
The output looks as follows:
```sh
[ INFO ] Dump mode was enabled
[ INFO ] <layer_1_name> layer processing
...
[ INFO ] <layer_X_name> layer processing
[ INFO ] Dump file path: <path_where_dump_will_be_saved>
[ INFO ] Execution successful
```
If you do not provide the `-i` key, the Cross Check Tool generates an input from normal distributed noise and saves
it in a multi-input file format with the filename `<path_to_xml>_input_layers_dump.txt` in the same folder as the Intermediate Representation (IR).
4. To check the overall accuracy and performance of inference on the CPU in FP32 precision against dumped results, run:
```sh
$python3 cross_check_tool.py -i <path_to_input_image_or_multi_input_file> \
-m <path_to_FP32_xml> \
-d CPU \
--load <path_to_dump> \
--layers all
```
The output looks as follows:
```sh
[ INFO ] Load mode was enabled
[ INFO ] IR for CPU : <path_to_FP32_xml>
[ INFO ] Loading blob from /localdisk/models/FP16/icv_squeezenet_v1.0.xml_GPU_dump.npz
[ INFO ] Statistics will be dumped for X layers: <layer_1_name>, <layer_2_name>, ... , <layer_X_name>
[ INFO ] Layer <layer_1_name> statistics
Max absolute difference : 0.0
Min absolute difference : 0.0
Max relative difference : 0.0
Min relative difference : 0.0
Min reference value : 0.0
Min absolute reference value : 0.0
Max reference value : 7.14844E-01
Max absolute reference value : 7.14844E-01
Min actual value : 0.0
Min absolute actual value : 0.0
Max actual value : 7.14844E-01
Max absolute actual value : 7.14844E-01
Device: -d CPU -load GPU
Status: OPTIMIZED_OUT OPTIMIZED_OUT
Layer type: Reshape Reshape
Real time, microsec: 0 0
Number of NAN: 0 0
Number of INF: 0 0
Number of ZERO: 609 699
...
<list_of_layer_statistics>
...
[ INFO ] Overall max absolute difference = 0.0
[ INFO ] Overall min absolute difference = 0.0
[ INFO ] Overall max relative difference = 0.0
[ INFO ] Overall min relative difference = 0.0
[ INFO ] Execution successful
```
### Multi-input and dump file format
Multi-input and dump file is a numpy compressed `.npz` file with hierarchy:
```sh
{
‘layer_name’: {
‘blob’: np.array([…])
‘pc’: {
‘device’: ‘device_name’,
‘real_time’: int_real_time_in_microseconds_from_plugin,
‘exec_type’: ‘exec_type_from_plugin’,
‘layer_type’: ‘layer_type_from_plugin’,
‘status’: ‘status_from_plugin’
}
},
‘another_layer_name’: {
‘blob’: np.array([…])
‘pc’: {
‘device’: ‘device_name’,
‘real_time’: int_real_time_in_microseconds_from_plugin,
‘exec_type’: ‘exec_type_from_plugin’,
‘layer_type’: ‘layer_type_from_plugin’,
‘status’: ‘status_from_plugin’
}
},
...
}
```
### Configuration file
There is an option to pass configuration file to plugin by providing
`--config` and/or `--reference_config` keys.
Configuration file is a text file with content of pairs of keys and values.
Structure of configuration file:
```sh
KEY VALUE
ANOTHER_KEY ANOTHER_VALUE,VALUE_1
```

11
tools/cross_check_tool/cross_check_tool.py
View File

@ -1,11 +0,0 @@
#!/usr/bin/python3
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import sys
from openvino.tools.cross_check_tool.cross_check_tool import main
if __name__ == "__main__":
sys.exit(main() or 0)

3
tools/cross_check_tool/openvino/__init__.py
View File

@ -1,3 +0,0 @@
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
__path__ = __import__('pkgutil').extend_path(__path__, __name__)

3
tools/cross_check_tool/openvino/tools/__init__.py
View File

@ -1,3 +0,0 @@
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

3
tools/cross_check_tool/openvino/tools/cross_check_tool/__init__.py
View File

@ -1,3 +0,0 @@
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

323
tools/cross_check_tool/openvino/tools/cross_check_tool/cross_check_tool.py
View File

@ -1,323 +0,0 @@
#!/usr/bin/python3
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import datetime
import logging as log
import sys
from collections import defaultdict
from typing import Union
import numpy as np
try:
from openvino.runtime import Core, Model, CompiledModel, InferRequest, Output, get_version
except Exception as e:
exception_type = type(e).__name__
print(f"The following error happened while importing OpenVINO Python API module:\n[ {exception_type} ] {e}")
sys.exit(1)
from openvino.tools.cross_check_tool.utils import get_config_dictionary, get_ops_list, print_output_ops, \
input_processing, accuracy_metrics, validate_args, build_parser, set_logger, find_out_cct_mode, \
print_all_over_the_net_metrics, update_global_accuracy_matrics, tensor_counters, performance_metrics, \
dump_output_file, load_dump, error_handling, print_inputs, set_verbosity, perf_counts_to_dump, load_profiling_info
###
# PLUGIN
###
@error_handling('plugin of \'{device}\' device config \'{config}\' loading')
def set_plugin_config(core: Core, device: str, config: str = None):
core.set_property(device, get_config_dictionary(config_file=config))
@error_handling('\'{cpu_ext}\' cpu extensions loading')
def set_cpu_extensions(core: Core, cpu_ext: str):
core.add_extension(cpu_ext)
def get_plugin(device: str, cpu_ext: str = None, config: str = None):
core = Core()
# log.info('{} plugin:\n API version ............ {}'.format(device, plugin.version), extra={'no_lvl': True})
set_plugin_config(core=core, device=device, config=config)
if cpu_ext and 'CPU' in device:
set_cpu_extensions(core=core, cpu_ext=cpu_ext)
return core
###
# MODEL
###
@error_handling('reading {xml_path} IR model')
def get_model(model_path: str, core: Core):
model = core.read_model(model=model_path)
# TODO: can we support it?
if model.is_dynamic():
raise Exception("Cross check tool doesn't support dynamic models for now.")
return model
@error_handling('compiling model for {device} device')
def get_compiled_model(core: Core, model: Model, device: str):
return core.compile_model(model=model, device_name=device)
@error_handling('creating infer request')
def get_infer_request(compiled_model: CompiledModel):
return compiled_model.create_infer_request()
@error_handling('output \'{output}\' addition for network from model \'{model}\'')
def get_model_copy_with_output(model: str, output: tuple, core: Core):
model_copy = get_model(model_path=model, core=core)
new_output = None
if output not in ['None', None]:
new_output = model_copy.add_outputs(output).pop()
return model_copy, new_output
@error_handling('getting model operations info')
def get_model_info(model: Model):
return model.get_ordered_ops(), model.inputs, model.outputs
def check_inputs_and_default_outputs_are_equal(model, ref_model):
if len(model.inputs) != len(ref_model.inputs):
raise Exception("Models have different number of inputs! Cannot cross check!")
if len(model.outputs) != len(ref_model.outputs):
raise Exception("Models have different number of outputs! Cannot cross check!")
for input, ref_input in zip(model.inputs, ref_model.inputs):
if input.any_name != ref_input.any_name:
raise Exception("Models have different inputs! Cannot cross check!")
for output, ref_output in zip(model.outputs, ref_model.outputs):
if output.any_name != ref_output.any_name:
raise Exception("Models have different outputs! Cannot cross check!")
def get_ops_intersection(ops, ref_ops):
ops_map = {node.friendly_name: node for node in ops}
operation_names = set(ops_map.keys())
ref_operation_names = set(node.friendly_name for node in ref_ops)
intersection_names = operation_names.intersection(ref_operation_names)
return [ops_map[intersection_name] for intersection_name in intersection_names]
def get_ops_union(ops, ref_ops):
ops_map = {}
for op, ref_op in zip(ops, ref_ops):
ops_map.update({op.friendly_name: op})
ops_map.update({ref_op.friendly_name: ref_op})
return ops_map.values()
###
# INFER
###
@error_handling('getting inference results for output: \'{output.any_name}\'')
def get_infer_results(infer_request: InferRequest, output: Output):
return infer_request.get_tensor(output).data
@error_handling('getting performance counts from infer request')
def get_profiling_info(infer_request: InferRequest, port: Output):
for pi in infer_request.profiling_info:
if pi.node_name == port.node.friendly_name:
return pi
@error_handling('processing inference on \'{device}\' device')
def infer(model: Model, core: Core, device: str, inputs: Union[list, dict], output=None):
compiled_model = get_compiled_model(core=core, model=model, device=device)
infer_request = get_infer_request(compiled_model)
infer_request.infer(inputs)
if output:
result = get_infer_results(infer_request, output)
prof_info = get_profiling_info(infer_request, output)
return result, prof_info
@error_handling('computing overall performance')
def overall_accuracy_check(model: str, ref_model: str, out_ops: list, ref_out_ops: list, inputs: list,
ref_inputs: list, core: Core, device: str, ref_core: Core, ref_device: str, layers: str,
num_of_iterations: int):
global_times, ref_global_times = [], []
if layers in ['None', None]:
model_copy, _ = get_model_copy_with_output(model=model, output=layers, core=core)
ref_model_copy, _ = get_model_copy_with_output(model=ref_model, output=layers, core=ref_core)
for i in range(num_of_iterations):
t1 = datetime.datetime.now()
infer(model=model_copy, core=core, device=device, inputs=inputs)
t2 = datetime.datetime.now()
infer(model=ref_model_copy, core=ref_core, device=ref_device, inputs=ref_inputs)
t3 = datetime.datetime.now()
global_times.append(t2 - t1)
ref_global_times.append(t3 - t2)
return global_times, ref_global_times
def one_ir_mode(args):
core = get_plugin(args.device, args.l, args.config)
model = get_model(model_path=args.model, core=core)
model_ops, model_inputs, model_outputs = get_model_info(model)
log.info(f'{args.device} vs {args.reference_device}')
log.info(f'The same IR on both devices: {args.model}')
out_ops = get_ops_list(model_ops, model_outputs, args.layers)
print_inputs(model_inputs)
print_output_ops(out_ops)
ref_core = get_plugin(args.reference_device, args.l, args.reference_config)
global_accuracy = []
inputs = input_processing(model_path=args.model, model_inputs=model_inputs, input_file=args.input)
global_times, ref_global_times = overall_accuracy_check(model=args.model, ref_model=args.model,
out_ops=out_ops, ref_out_ops=out_ops,
inputs=inputs, ref_inputs=inputs, core=core,
device=args.device, ref_core=ref_core,
ref_device=args.reference_device, layers=args.layers,
num_of_iterations=args.num_of_iterations)
for op in out_ops:
log.info(f'Layer {op.friendly_name} statistics')
for i in range(op.get_output_size()):
if op.get_output_size() > 1:
log.info(f'Port {i}: ')
model_copy, new_output = get_model_copy_with_output(model=args.model, output=(op.friendly_name, i), core=core)
out_tensor, pc = infer(model=model_copy, core=core, device=args.device, inputs=inputs, output=new_output)
ref_out_tensor, ref_pc = infer(model=model_copy, core=ref_core, device=args.reference_device, inputs=inputs, output=new_output)
a_m = accuracy_metrics(out_tensor, ref_out_tensor)
performance_metrics(args.device, pc, args.reference_device, ref_pc)
tensor_counters(out_tensor, ref_out_tensor)
global_accuracy = update_global_accuracy_matrics(global_accuracy=global_accuracy, current_accuracy=a_m)
print_all_over_the_net_metrics(global_times=global_times, ref_global_times=ref_global_times,
global_accuracy=global_accuracy)
def two_ir_mode(args):
core = get_plugin(args.device, args.l, args.config)
ref_core = get_plugin(args.reference_device, args.l, args.reference_config)
model = get_model(model_path=args.model, core=core)
model_ops, model_inputs, model_outputs = get_model_info(model)
ref_model = get_model(model_path=args.reference_model, core=ref_core)
ref_model_ops, _, _ = get_model_info(ref_model)
check_inputs_and_default_outputs_are_equal(model, ref_model)
log.info(f'{args.device} vs {args.reference_device}')
log.info(f'IR for {args.device} : {args.model}')
log.info(f'IR for {args.reference_device} : {args.reference_model}')
if args.reference_layers:
out_ops = get_ops_list(model_ops, model_outputs, args.layers)
ref_out_ops = get_ops_list(ref_model_ops, model_outputs, args.reference_layers)
if len(out_ops) != len(ref_out_ops):
raise Exception("Number of layers to compare against should be equal!")
else:
ref_out_ops = out_ops = get_ops_list(get_ops_intersection(model_ops, ref_model_ops), model_outputs, args.layers)
print_inputs(model_inputs)
print_output_ops(get_ops_union(out_ops, ref_out_ops))
inputs = input_processing(model_path=args.model, model_inputs=model_inputs, input_file=args.input)
global_accuracy = []
global_times, ref_global_times = overall_accuracy_check(model=args.model, ref_model=args.reference_model,
out_ops=out_ops, ref_out_ops=out_ops,
inputs=inputs, ref_inputs=inputs, core=core,
device=args.device, ref_core=ref_core,
ref_device=args.reference_device, layers=args.layers,
num_of_iterations=args.num_of_iterations)
for op, ref_op in zip(out_ops, ref_out_ops):
if op.friendly_name == ref_op.friendly_name:
log.info(f'Layer {op.friendly_name} statistics')
else:
if op.get_output_size() != ref_op.get_output_size():
log.warning(f"Skipping {op.friendly_name} vs {ref_op.frinedly_name} comparison due to different number of outputs!")
continue
log.info(f'Layer {op.friendly_name} vs {ref_op.friendly_name} statistics')
for i in range(op.get_output_size()):
if op.get_output_size() > 1:
log.info(f'Port {i}: ')
model_copy, new_output = get_model_copy_with_output(model=args.model, output=(op.friendly_name, i), core=core)
ref_model_copy, ref_new_output = get_model_copy_with_output(model=args.reference_model, output=(ref_op.friendly_name, i), core=ref_core)
out_tensor, pc = infer(model=model_copy, core=core, device=args.device, inputs=inputs, output=new_output)
ref_out_tensor, ref_pc = infer(model=ref_model_copy, core=ref_core, device=args.reference_device,
inputs=inputs, output=ref_new_output)
a_m = accuracy_metrics(out_tensor, ref_out_tensor)
performance_metrics(args.device, pc, args.reference_device, ref_pc)
tensor_counters(out_tensor, ref_out_tensor)
global_accuracy = update_global_accuracy_matrics(global_accuracy=global_accuracy, current_accuracy=a_m)
print_all_over_the_net_metrics(global_times=global_times, ref_global_times=ref_global_times,
global_accuracy=global_accuracy)
def dump_mode(args):
core = get_plugin(args.device, args.l, args.config)
model = get_model(model_path=args.model, core=core)
model_ops, model_inputs, model_outputs = get_model_info(model)
out_ops = get_ops_list(model_ops, model_outputs, args.layers)
inputs = input_processing(args.model, model_inputs, args.input)
dump_dict = defaultdict(list)
for op in out_ops:
for i in range(op.get_output_size()):
if op.get_output_size() > 1:
log.info(f'Layer {op.friendly_name}, port {i} processing')
else:
log.info(f'Layer {op.friendly_name} processing')
model_copy, new_output = get_model_copy_with_output(model=args.model, output=(op.friendly_name, i), core=core)
out_tensor, pc = infer(model=model_copy, core=core, device=args.device, inputs=inputs, output=new_output)
dump_dict[op.friendly_name].append(np.array({'tensor': out_tensor, 'pc': perf_counts_to_dump(pc)}))
dump_dict["device"] = args.device
dump_output_file(args.model + '_' + args.device + '_dump.npz', dump_dict)
def load_mode(args):
core = get_plugin(args.device, args.l, args.config)
log.info(f'IR for {args.device} : {args.model}')
log.info(f'Loading tensors from {args.load}')
model = get_model(model_path=args.model, core=core)
model_ops, model_inputs, model_outputs = get_model_info(model)
out_ops = get_ops_list(model_ops, model_outputs, args.layers)
print_inputs(model_inputs)
print_output_ops(out_ops)
inputs = input_processing(args.model, model_inputs, args.input)
global_accuracy = []
loaded = load_dump(args.load)
for op in out_ops:
if op.friendly_name in loaded:
log.info(f'Layer {op.friendly_name} statistics')
else:
log.info(f'Statistics for layer \'{op.friendly_name}\' was not dumped. Skipping this layer.')
continue
for i in range(op.get_output_size()):
if op.get_output_size() > 1:
log.info(f'Port {i}: ')
model_copy, new_output = get_model_copy_with_output(model=args.model, output=(op.friendly_name, i), core=core)
out_tensor, pc = infer(model=model_copy, core=core, device=args.device, inputs=inputs, output=new_output)
ref_out_tensor, ref_pc = loaded[op.friendly_name][i]['tensor'], load_profiling_info(loaded[op.friendly_name][i]['pc'])
a_m = accuracy_metrics(out_tensor, ref_out_tensor)
performance_metrics(args.device, pc, loaded["device"], ref_pc)
tensor_counters(out_tensor, ref_out_tensor)
global_accuracy = update_global_accuracy_matrics(global_accuracy=global_accuracy, current_accuracy=a_m)
print_all_over_the_net_metrics(global_accuracy=global_accuracy)
def main():
set_logger(log.DEBUG)
args = validate_args(build_parser().parse_args())
log.info(f'OpenVINO:\n API version ............ {get_version()}', extra={'no_lvl': True})
set_verbosity(args.verbosity)
mode = find_out_cct_mode(args)
if mode == 1:
log.info('Cross check with one IR was enabled')
one_ir_mode(args)
elif mode == 2:
log.info('Cross check with two IRs was enabled')
two_ir_mode(args)
elif mode == 3:
log.info('Dump mode was enabled')
dump_mode(args)
elif mode == 4:
log.info('Load mode was enabled')
load_mode(args)
log.info("Execution successful")
if __name__ == '__main__':
main()

567
tools/cross_check_tool/openvino/tools/cross_check_tool/utils.py
View File

@ -1,567 +0,0 @@
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
import argparse
import logging as log
import os
import sys
import traceback
from pathlib import Path
from openvino.runtime import Layout, ProfilingInfo, Output
from openvino.runtime.utils.types import get_dtype
try:
import cv2
except Exception as e:
log.error(f"Can not import OpenCV Python package.\nPlease install required python packages by running:\n"
f"pip3 install -r requirements.txt\n\n Original error message: {e}")
sys.exit(1)
try:
import numpy as np
except Exception as e:
log.error(f"Can not import numpy python package.\nPlease install required python packages by running:\n"
f"pip3 install -r requirements.txt\n\n Original error message: {e}")
sys.exit(1)
verbosity = False
def set_verbosity(flag: bool):
global verbosity
verbosity = flag
###
# USER INTERACTION
###
class LvlFormatter(log.Formatter):
usual = '[ %(levelname)s ] %(msg)s'
format_dict = {
'no_lvl': '%(msg)s',
log.DEBUG: '[ %(asctime)s ] [ %(levelname)s ] [ %(module)s:%(lineno)d ] %(msg)s',
log.INFO: usual, log.WARNING: usual, log.ERROR: usual, log.CRITICAL: usual
}
def __init__(self, lvl, fmt=None):
log.Formatter.__init__(self, fmt)
self.lvl = lvl
def format(self, record: log.LogRecord):
if self.lvl == 'DEBUG':
self._style._fmt = self.format_dict[log.DEBUG]
else:
self._style._fmt = self.format_dict[record.levelno]
if 'no_lvl' in record.__dict__.keys() and record.__dict__['no_lvl']:
self._style._fmt = self.format_dict['no_lvl']
return log.Formatter.format(self, record)
def set_logger(lvl: str):
logger = log.getLogger()
logger.setLevel(lvl)
handler = log.StreamHandler(sys.stdout)
handler.setFormatter(LvlFormatter(lvl))
logger.addHandler(handler)
def error_handling(desc: str):
"""
Error handler that prints description formatted with keyword arguments in case of exception
:param desc: description for an error
:return: decorator
"""
def decorator(func):
def try_except_func(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
exception_type = type(e).__name__
log.error(f"The following error happened while {desc.format(**kwargs)}:\n[ {exception_type} ] {e}")
global verbosity
if verbosity:
traceback.print_tb(tb=e.__traceback__, file=sys.stdout)
sys.exit(1)
return try_except_func
return decorator
class ExistingFileAction(argparse.Action):
"""
Expand user home directory paths and convert relative-paths to absolute.
"""
def __call__(self, parser, namespace, values, option_string=None):
if values is not None:
if not os.path.isfile(values):
log.error(f"File was not found: {values}")
sys.exit(1)
setattr(namespace, self.dest, values)
class ExistingDirAction(argparse.Action):
"""
Expand user home directory paths and convert relative-paths to absolute.
"""
def __call__(self, parser, namespace, values, option_string=None):
if values is not None:
if not os.path.isdir(values):
log.error(f"Directory was not found: {values}")
sys.exit(1)
setattr(namespace, self.dest, values)
def build_parser():
parser = argparse.ArgumentParser(
prog='Cross Check Tool',
description='Cross Check Tool is a console application that enables comparing accuracy and provides performance'
' metrics',
usage='\n' + '-' * 62 +
'\nFor cross precision check provide two IRs \n'
'(mapping files may be needed) run:'
'\npython3 cross_check_tool.py \\'
'\n--input path/to/file/describing/input \\'
'\n--model path/to/model/*.xml \\'
'\n--device device_for_model \\'
'\n--reference_model path/to/reference_model/*.xml \\'
'\n--reference_device reference_device_for_model \n'
+ '-' * 62 +
'\nFor cross device check with one precision provide one IR run:'
'\npython3 cross_check_tool.py \\'
'\n--input path/to/file/describing/input \\'
'\n--model path/to/model/*.xml \\'
'\n--device device_for_model \\'
'\n--reference_device reference_device_for_model \n'
+ '-' * 62 +
'\nFor dumping tensors and performance counters run:'
'\npython3 cross_check_tool.py \\'
'\n--input path/to/file/describing/input \\'
'\n--model path/to/model/*.xml \\'
'\n--device device_for_model \\'
'\n--dump\n'
+ '-' * 62 +
'\nFor check inference against dumped results run:'
'\npython3 cross_check_tool.py \\'
'\n--input path/to/file/describing/input \\'
'\n--model path/to/model/*.xml \\'
'\n--device device_for_model \\'
'\n--load path/to/dump/file/* \n'
+ '-' * 62 +
'\nFor all layers check provide:\n'
'--layers=\'all\' \n'
'For specific number of layers check provide:\n'
'--layers=\'layer_name,another_layer_name,...,last_layer_name\'\n'
+ '-' * 62 +
'\nIf --input is empty CCT generates input(s) from normal\n'
'distribution and dumps this input to a file\n'
+ '-' * 62
)
model = parser.add_argument_group('Model specific arguments')
model.add_argument('--input', '-i', type=str,
help='Path to an input image file or multi-input file to infer. Generates input(s) from normal '
'distribution if empty')
# model.add_argument('--batch', '-b', type=int, help='Overrides batch size. Default is inherited from model')
model.add_argument('--model', '-m', type=str, action=ExistingFileAction,
help='Path to an .xml file that represents the first IR of the trained model to infer.')
model.add_argument('--reference_model', '-ref_m', type=str, action=ExistingFileAction,
help='Path to an .xml file that represents the second IR to compare the metrics. '
'Uses --model if empty')
# TODO: allow to pass layer type
model.add_argument('--layers', '-layers', type=str, default=None,
help='Defines layers to check. Options: all, None - for output layers check, list of '
'comma-separated layer names to check. Default value is None.')
model.add_argument('-ref_layers', '--reference_layers', type=str, default=None,
help='Defines layers to check in referece model. Options: all, None - for output layers check, list of '
'comma-separated layer names to check. If not specified the same layers will be processed as in --layers parameter.')
plugin = parser.add_argument_group('Plugin specific arguments')
plugin.add_argument('--plugin_path', '-pp', type=str, action=ExistingDirAction, help='Path to a plugin folder.')
plugin.add_argument('--device', '-d', type=str, required=True,
help='The first target device to infer the model specified with the -m or --model option. '
'CPU, GPU or GNA are acceptable.')
plugin.add_argument('--config', '-conf', type=str, action=ExistingFileAction,
help='Path to config file for -d or -device device plugin')
plugin.add_argument('--reference_device', '-ref_d', type=str,
help='The second target device to infer the model and compare the metrics. '
'CPU, GPU or GNA are acceptable.')
plugin.add_argument('--reference_config', '-ref_conf', type=str, action=ExistingFileAction,
help='Path to config file for -ref_d or -reference_device device plugin')
plugin.add_argument('-l', type=str, action=ExistingFileAction,
help='Required for (CPU)-targeted custom layers. Comma separated paths to a shared'
' libraries with the kernels implementation.')
modes = parser.add_argument_group('CCT mode arguments')
# TODO eps? nobody uses it
modes.add_argument('--dump', help='Enables tensors statistics dumping', action='store_true', default=False)
modes.add_argument('--load', type=str, action=ExistingFileAction, help='Path to a file to load tensors from')
model.add_argument('--num_of_iterations', '-ni', type=int, default=50,
help='Number of iterations to collect all over the net performance')
parser.add_argument('-v', '--verbosity', action='store_true', default=False,
help='Increase output verbosity')
return parser
@error_handling('validating arguments passed to cross_check_tool.py')
def validate_args(args):
# input check
if args.input is None:
log.info('No input was provided by --input/-i. Generate input from noise')
# model check
if args.model is None and args.reference_model is None:
raise Exception(
"Parameters --model/-m and --reference_model/-ref_m are empty. At least one of them is required")
elif args.model is None and args.reference_model:
args.model = args.reference_model
if args.model == args.reference_model:
args.reference_model = None
if args.model != args.reference_model and args.reference_model is not None and (args.layers is None or \
args.reference_layers is None):
log.warning('Check over two different IRs was enabled. In case if layer names in these two IRs are different, '
'please provide both -layers and --reference_layers to compare against.')
# device check
if args.device is None and args.reference_device is None:
raise Exception("Parameters -device/-d and -reference_device/-ref_d are not set. Can not proceed."
"\nFor more details use -h option")
if args.reference_device is None and args.reference_model is None and not args.dump and args.load is None:
raise Exception("Please provide --reference_model/-ref_m to compare executions on different devices."
"\nAnother option is to provide --dump key to dump all execution info on one device."
"\nOr provide --load key to compare execution on device with dumped info"
"\nFor more details use -h option")
if args.device is None:
args.device = args.reference_device
args.reference_device = None
# dump and load check
if args.dump and args.load is not None:
raise Exception("Cross Check Tool does not support both loading and dumping modes to be enabled. "
"Choose one of them and proceed")
if args.model is not None and args.reference_model is not None and args.dump or \
args.device is not None and args.reference_device is not None and args.dump:
raise Exception("Cross Check Tool does support dumping mode to be enabled only for one model on one device"
"\nFor more details use -h option")
if args.model is not None and args.reference_model is not None and args.load is not None or \
args.device is not None and args.reference_device is not None and args.load is not None:
raise Exception("Cross Check Tool does support loading mode to be enabled for one model on one device against a"
" dumped file\nFor more details use -h option")
return args
def find_out_cct_mode(args):
"""
1 -- one IR mode
2 -- two IRs mode
3 -- dump mode
4 -- load mode
"""
# dump mode
if args.dump and args.model is not None and args.device is not None and \
args.reference_model is None and args.reference_device is None:
return 3
# load mode
if args.load is not None and args.model is not None and args.device is not None and args.reference_device is None:
return 4
# two IR mode
if args.model is not None and args.reference_model is not None:
return 2
# one IR mode
if args.model is not None and args.reference_model is None:
return 1
raise Exception('Unknown Cross Check Tool CLI configuration.\nFor more details use -h option')
def print_inputs(inputs: list):
word = 'inputs' if len(inputs) > 1 else 'input'
log.info(f"{len(inputs)} {word} detected: {', '.join(input.any_name for input in inputs)}")
def print_output_ops(output_ops: list):
layers = 'layers' if len(output_ops) > 1 else 'layer'
log.info(f"Statistics will be dumped for {len(output_ops)} {layers}: {', '.join(op.friendly_name for op in output_ops)}")
###
# PLUGIN
###
@error_handling('parsing config file for plugin: \'{config_file}\'')
def get_config_dictionary(config_file):
config = {'PERF_COUNT': 'YES'}
if not config_file:
return config
with open(config_file) as f:
config_line = f.readline()
key = config_line.split()[0]
value = config_line[len(key):].strip()
config[key] = value
return config
###
# INPUTS
###
def read_multi_input_file(input_file: str, model_inputs: list):
npz = np.load(input_file, allow_pickle=True)
files = npz.files
dump = []
for model_input in model_inputs:
if model_input.any_name not in files:
raise Exception(f"Can not find input data for input {model_input.any_name} in multi-input file {input_file}.\n"
f"Input data was provided for layers: {', '.join(files)}\n"
f"Network inputs: {', '.join(input.any_name for input in model_inputs)}")
if 'tensor' in npz[model_input.any_name].item(0):
just_tensor = npz[model_input.any_name].item(0)['tensor']
input_shape = list(model_input.shape)
log.info(f'Layer {model_input.any_name} shape = {input_shape}, input tensor from multi-input file shape = {just_tensor.shape}')
try:
reshaped_tensor = np.reshape(just_tensor, input_shape)
except:
raise Exception(f'Can not reshape input tensor from multi-input file for layer {model_input.any_name} to shape {input_shape}')
dump.append(reshaped_tensor)
else:
raise Exception(
f'Can not find \'tensor\' parameter for input {model_input.any_name} in input file {input_file}')
return dump
def is_chw_input(input):
if input.node.layout == Layout("NCHW") or input.node.layout == Layout("CHW"):
return True
shape = input.shape
if len(shape) == 4:
return shape[1] == 3
if len(shape) == 3:
return shape[0] == 3
return False
def get_input_sizes(input):
if is_chw_input(input):
shape = input.shape
if len(shape) == 3:
return shape[1], shape[2]
else:
return shape[2], shape[3]
else:
shape = input.shape
if len(shape) < 3 or len(shape) > 4:
raise Exception('Can not interpret input shape as image')
if len(shape) == 3:
return shape[0], shape[1]
else:
return shape[1], shape[2]
@error_handling('reading --input/-i by OpenCV python module. OpenCV version: {}. '
'It may happen due to wrong input image format'.format(cv2.__version__))
def read_image_file(image_file: str, model_input: Output):
image_file = str(image_file)
log.info(f'Prepare image {image_file}')
image = cv2.imread(image_file)
if image is None:
raise Exception('Can not read input image ' + image_file)
h, w = get_input_sizes(model_input)
image = cv2.resize(image, (w, h))
if is_chw_input(model_input):
image = image.transpose((2, 0, 1)) # Change data layout from HWC to CHW
if len(model_input.shape) == 4:
image = np.expand_dims(image, 0) # Add batch dimension
return image
def get_random_inputs(model_inputs, model_path):
inputs = [np.clip(np.random.normal(0.5, 0.1, size=list(input.shape)), 0, 1) for input in model_inputs]
dump_output_file(model_path + '_random_input_dump.npz', {model_inputs[i].any_name: {'tensor': inputs[i]} for i in range(len(model_inputs))})
return inputs
def read_binary_file(bin_file, model_input):
log.info(f"Prepare binary file {str(bin_file)}")
binary_file_size = os.path.getsize(bin_file)
tensor_size = model_input.tensor.size
if tensor_size != binary_file_size:
raise Exception(f"File {bin_file} contains {binary_file_size} bytes but model expects {tensor_size}")
return np.reshape(np.fromfile(bin_file, get_dtype(model_input.element_type)), list(model_input.shape))
def input_processing(model_path: str, model_inputs: list, input_file: str):
if input_file is None:
return get_random_inputs(model_inputs, model_path)
else:
inputs = input_file.split(',')
input_names = [input.any_name for input in model_inputs]
input_data = {}
for i in range(min(len(inputs), len(model_inputs))):
splited = inputs[i].rsplit(':', maxsplit=1)
if len(splited) == 0:
raise Exception(f"Can't parse {'input_file'} input parameter!")
tensor_name = None
if len(splited) == 1:
tensor_name = input_names[i]
else:
tensor_name = splited.pop(0)
if tensor_name not in input_names:
raise Exception(f"Input with name {tensor_name} doesn't exist in the model!")
path = Path(splited.pop())
if path.exists() and path.is_file():
IMAGE_EXTENSIONS = ['.jpeg', '.jpg', '.png', '.bmp']
BINARY_EXTENSIONS = ['.bin']
NUMPY_EXTENSIONS = ['.npy', '.npz']
current_input = model_inputs[input_names.index(tensor_name)]
if path.suffix.lower() in IMAGE_EXTENSIONS:
input_data[tensor_name] = read_image_file(path, current_input)
elif path.suffix.lower() in BINARY_EXTENSIONS:
input_data[tensor_name] = read_binary_file(path, current_input)
elif path.suffix.lower() in NUMPY_EXTENSIONS:
return read_multi_input_file(path, model_inputs)
elif path.is_dir():
raise Exception(f"Path `{path}` is a directory! Provide full path to an input file!")
elif not path.exists():
raise Exception(f"Input file `{path}` doesn't exist!")
return input_data
def accuracy_metrics(tensor, ref_tensor):
if tensor.size != ref_tensor.size:
raise Exception(f'Different number of elements in tensors {tensor.size} and {ref_tensor.size}. Can not compare')
abs_diff = np.absolute(tensor - ref_tensor)
np.seterr(divide='ignore', invalid='ignore')
rel_diff = np.divide(abs_diff, np.min(abs_diff) if np.min(abs_diff) != 0 else 1e-20)
metrics = [
('Max absolute difference', np.max(abs_diff)),
('Min absolute difference', np.min(abs_diff)),
('Max relative difference', np.max(rel_diff)),
('Min relative difference', np.min(rel_diff)),
('Min reference value', np.min(ref_tensor)),
('Min absolute reference value', np.min(np.abs(ref_tensor))),
('Max reference value', np.max(ref_tensor)),
('Max absolute reference value', np.max(np.abs(ref_tensor))),
('Min actual value', np.min(tensor)),
('Min absolute actual value', np.min(np.abs(tensor))),
('Max actual value', np.max(tensor)),
('Max absolute actual value', np.max(np.abs(tensor)))
]
for key, value in metrics:
if len(str(value)) > 5:
log.info(f'{key:>35} : {value:.5E}', extra={'no_lvl': True})
else:
log.info(f'{key:>35} : {value}', extra={'no_lvl': True})
return {metric: value for metric, value in metrics}
def performance_metrics(device, pc, ref_device, ref_pc):
compare = [
('Device', '-d ' + device, '-ref_d ' + ref_device),
('Status', str(pc.status), str(ref_pc.status)),
('Layer type', pc.node_type, ref_pc.node_type),
('Real time, microsec', str(pc.real_time), str(ref_pc.real_time))
]
for metric, actual, reference in compare:
log.info(f'{metric:>35}: {actual:>16} {reference:>16}', extra={'no_lvl': True})
def tensor_counters(tensor, ref_tensor):
counters = [
('Number of NAN', np.sum(np.isnan(tensor)), np.sum(np.isnan(ref_tensor))),
('Number of INF', np.sum(np.isinf(tensor)), np.sum(np.isinf(ref_tensor))),
('Number of ZERO', tensor.size - np.count_nonzero(tensor),
ref_tensor.size - np.count_nonzero(ref_tensor))
]
for metric, actual, reference in counters:
log.info(f'{metric:>35}: {actual:>16} {reference:>16}', extra={'no_lvl': True})
def update_global_accuracy_matrics(global_accuracy: list, current_accuracy: dict):
metrics = [
('Max absolute difference', lambda x, y: max(x, y)),
('Min absolute difference', lambda x, y: min(x, y)),
('Max relative difference', lambda x, y: max(x, y)),
('Min relative difference', lambda x, y: min(x, y))]
for metric, formula in metrics:
global_metric = [item for item in global_accuracy if item[0] == metric]
if len(global_metric) == 1:
g_metric, g_value = global_metric[0]
global_accuracy.remove(global_metric[0])
global_accuracy.append((metric, formula(g_value, current_accuracy[metric])))
else:
global_accuracy.append((metric, current_accuracy[metric]))
return global_accuracy
def print_all_over_the_net_metrics(global_accuracy: list, global_times: list = None,
ref_global_times: list = None):
if global_times is not None and ref_global_times is not None and len(global_times) and len(ref_global_times):
log.info('-' * 70, extra={'no_lvl': True})
log.info(f'{"Overall performance, microseconds":>35}: '
f'{global_times[len(global_times) // 2].microseconds:>16,.5E} '
f'{ref_global_times[len(ref_global_times) // 2].microseconds:>16,.5E}',
extra={'no_lvl': True})
log.info('-' * 70, extra={'no_lvl': True})
for metric, value in global_accuracy:
log.info(f"Overall {metric.lower()} = {value}")
###
# MAPPING
###
def get_ops_list(all_ops: list, outputs: list, layers: str):
if layers is not None and layers != 'None':
if layers == 'all':
return [op for op in all_ops if op.get_type_name() not in ['Constant', 'Result', 'Parameter']]
else:
all_ops_map = {op.friendly_name: op for op in all_ops}
user_ops = [layer.strip() for layer in layers.split(',')]
new_outputs = []
for user_op in user_ops:
if user_op not in all_ops_map:
raise Exception(f"Operation with name `{user_op}` doesn't exist in the model")
else:
new_outputs.append(all_ops_map[user_op])
return new_outputs
else:
return [output.node for output in outputs]
def perf_counts_to_dump(prof_info):
perf_count = {}
perf_count["status"] = prof_info.status
perf_count["real_time"] = prof_info.real_time
perf_count["cpu_time"] = prof_info.cpu_time
perf_count["exec_type"] = prof_info.exec_type
perf_count["node_type"] = prof_info.node_type
return perf_count
def load_profiling_info(pi_dumped):
prof_info = ProfilingInfo()
for property, value in pi_dumped.items():
setattr(prof_info, property, value)
return prof_info
###
# FILES
###
def dump_output_file(output_file, dump_dict):
np.savez_compressed(output_file, **dump_dict)
log.info(f'Dump file path: {output_file}')
def load_dump(file_to_load: str):
npz = np.load(file_to_load, allow_pickle=True)
dump = {file: [npz[file].item(i).item(0) for i in range(npz[file].size)] for file in npz if file != "device"}
dump["device"] = npz["device"].item(0)
return dump

2
tools/cross_check_tool/requirements.txt
View File

@ -1,2 +0,0 @@
numpy>=1.16.6
opencv-python>=4.5

40
tools/cross_check_tool/setup.py
View File

@ -1,40 +0,0 @@
#!/usr/bin/env python3
# Copyright (C) 2018-2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
"""
Use this script to create a wheel with OpenVINO™ Cross Check Tool:
$ python setup.py sdist bdist_wheel
"""
from pathlib import Path
from setuptools import setup, find_packages
SETUP_DIR = Path(__file__).resolve().parent
def read_text(path):
return (SETUP_DIR / path).read_text()
setup(
name='cross_check_tool',
version='0.0.0',
author='Intel® Corporation',
license='OSI Approved :: Apache Software License',
author_email='openvino_pushbot@intel.com',
url='https://github.com/openvinotoolkit/openvino',
description='OpenVINO™ Cross Check Tool package',
entry_points={
'console_scripts': [
'cross_check_tool = openvino.tools.cross_check_tool.cross_check_tool:main'],
},
classifiers=[
'Programming Language :: Python :: 3',
'OSI Approved :: Apache Software License',
'Operating System :: OS Independent',
],
packages=find_packages(),
install_requires=read_text('requirements.txt'),
python_requires='>=3.7',
)