IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

[IE TESTS][CONFORMANCE] Update conformance ReadMe files (#9374)

Browse Source 
* [IE TESTS][CONFORMANCE] Update conformance ReadMe files

* Fix links
This commit is contained in:
Irina Efode 2021-12-23 13:00:41 +03:00 committed by GitHub
parent f316801ccd
commit 7618fc8752
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 33 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
src/tests/functional/plugin/conformance/test_runner/README.md
View File

@ -3,6 +3,10 @@
## Description ## Description
Conformance suit is a set of tests with parameters independent from plug-in specific and limitations. It contains: Conformance suit is a set of tests with parameters independent from plug-in specific and limitations. It contains:
* `ReadIR`. Allow to read IRs from folders recursive, infer it and compare results with reference. * `ReadIR`. Allow to read IRs from folders recursive, infer it and compare results with reference.
* `OpImplCheckTest`. Allow to check operation plugin implementation status (`Implemented`/`Not implemented`).
**NOTE**: This test suite is in active development. The implementation status based on all other test results.
*
## How to build ## How to build
Run the following command in build directory: Run the following command in build directory:
@ -15,21 +19,36 @@ Run the following command in build directory:
make conformanceTests make conformanceTests
``` ```
## How to run ## How to generate Conformance operation set
Run the following commands:
1. Clone [`Open Model Zoo repo`](https://github.com/openvinotoolkit/open_model_zoo)
2. Download all possible models using [Downloader tool](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/downloader.py) from
the repo.
3. Convert downloaded models to IR files using [Converter tool](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/converter.py) from the repo.
4. Run [Subgraph dumper](./../subgraphs_dumper/README.md) to collect unique operation set from the models.
> **NOTE**:
>
> The conformance suite run in internal CI checks is based on `2021.4` release. **Please, use `Model optimizer`, 'Subgraph dumper' and `Open Model Zoo` tools
> from `2021.4`.**
## How to run operation conformance suite
The target is able to take the following command-line arguments: The target is able to take the following command-line arguments:
* `-h` prints target command-line options with description. * `-h` prints target command-line options with description.
* `--device` specifies target device. * `--device` specifies target device.
* `--input_folders` specifies folders with IRs to run. The separator is `,`. * `--input_folders` specifies folders with IRs to run. The separator is `,`.
* `--plugin_lib_name` is name of plugin library. The example is ov_intel_cpu_plugin. Use only with unregistered in IE Core devices. * `--plugin_lib_name` is name of plugin library. The example is ov_intel_cpu_plugin. Use only with unregistered in IE Core devices.
* `--disable_test_config` allows to ignore all skipped tests with the exception of `DISABLED_` prefix using. * `--disable_test_config` allows to ignore all skipped tests with the exception of `DISABLED_` prefix using.
* `--skip_config_path` allows to specify paths to files contain regular expressions list to skip tests. * `--skip_config_path` allows to specify paths to files contain regular expressions list to skip tests. [Examples](./op_conformance_runner/skip_configs)
* `--config_path` allows to specify path to file contains plugin config. [Example](./op_conformance_runner/config/config_example.txt)
* `--extend_report` allows not to re-write device results to the report (add results of this run to the existing). Mutually exclusive with --report_unique_name. * `--extend_report` allows not to re-write device results to the report (add results of this run to the existing). Mutually exclusive with --report_unique_name.
* `--report_unique_name` allows to save report with unique name (report_pid_timestamp.xml). Mutually exclusive with --extend_report. * `--report_unique_name` allows to save report with unique name (report_pid_timestamp.xml). Mutually exclusive with --extend_report.
* `--save_report_timeout` allows to try to save report in cycle using timeout (in seconds). * `--save_report_timeout` allows to try to save report in cycle using timeout (in seconds).
* `--output_folder` Paths to the output folder to save report. * `--output_folder` Paths to the output folder to save report.
* All `gtest` command-line parameters * All `gtest` command-line parameters
The result of execution is `report.xml` file. It demonstrates tests statistic like pass rate, passed, crashed, skipped and failed tests per operation for The result of execution is `report.xml` file. It demonstrates tests statistic like pass rate, passed, crashed, skipped failed tests and plugin implementation
per
operation for
devices. devices.
> **NOTE**: > **NOTE**:
@ -37,6 +56,8 @@ devices.
> Using of GTest parallel tool to run `conformanceTests` helps to report crashed tests and collect correct statistic > Using of GTest parallel tool to run `conformanceTests` helps to report crashed tests and collect correct statistic
> after unexpected crashes. > after unexpected crashes.
> >
> Use [`Gtest parallel`](https://github.com/google/gtest-parallel) from official repository with [this fix](https://github.com/google/gtest-parallel/pull/76).
>
> The example of usage is: > The example of usage is:
> ``` > ```
> python3 gtest_parallel.py /path/to/openvino/bin/intel64/Debug/conformanceTests -d . > python3 gtest_parallel.py /path/to/openvino/bin/intel64/Debug/conformanceTests -d .
@ -53,9 +74,15 @@ devices.
> python3 merge_xmls.py --input_folders=/path/to/temp_output_report_folder --output_folder=/path/to/output_report_folder --output_filename=report_aggregated > python3 merge_xmls.py --input_folders=/path/to/temp_output_report_folder --output_folder=/path/to/output_report_folder --output_filename=report_aggregated
> ``` > ```
## How to build operation coverage report ## How to create operation conformance report
Run [the script](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/summarize.py) to generate `html` report. Run [the script](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/summarize.py) to generate `html` report.
The example of using the script is: The example of using the script is:
``` ```
python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/ python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/
``` ```
> **NOTE**:
>
> Please, do not forget to copy [styles folder](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/template) to the output directory. It
> helps to provide report with the filters and other usable features.
Report contains statistic based on conformance results and filter fields at the top of the page.

2
src/tests/functional/plugin/conformance/test_runner/conformance_infra/include/gflag_config.hpp
View File

@ -26,7 +26,7 @@ static const char save_report_timeout_message[] = "Optional. Allow to try to sav
"Default value is 60 seconds"; "Default value is 60 seconds";
static const char skip_config_path_message[] = "Optional. Allows to specify paths to files contain regular expressions list to skip tests. " static const char skip_config_path_message[] = "Optional. Allows to specify paths to files contain regular expressions list to skip tests. "
"Delimiter is `,` symbol. Default value is empty string."; "Delimiter is `,` symbol. Default value is empty string.";
static const char config_path_message[] = "Optional. Allows to specify path to files contain plugin config. " static const char config_path_message[] = "Optional. Allows to specify path to file contains plugin config. "
"Default value is empty string."; "Default value is empty string.";