From 7618fc8752a593350a2b5b1b344969f1e282abe8 Mon Sep 17 00:00:00 2001 From: Irina Efode Date: Thu, 23 Dec 2021 13:00:41 +0300 Subject: [PATCH] [IE TESTS][CONFORMANCE] Update conformance ReadMe files (#9374) * [IE TESTS][CONFORMANCE] Update conformance ReadMe files * Fix links --- .../plugin/conformance/test_runner/README.md | 37 ++++++++++++++++--- .../include/gflag_config.hpp | 2 +- 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/src/tests/functional/plugin/conformance/test_runner/README.md b/src/tests/functional/plugin/conformance/test_runner/README.md index 0d6ba495df6..93160a8f617 100644 --- a/src/tests/functional/plugin/conformance/test_runner/README.md +++ b/src/tests/functional/plugin/conformance/test_runner/README.md @@ -3,6 +3,10 @@ ## Description Conformance suit is a set of tests with parameters independent from plug-in specific and limitations. It contains: * `ReadIR`. Allow to read IRs from folders recursive, infer it and compare results with reference. +* `OpImplCheckTest`. Allow to check operation plugin implementation status (`Implemented`/`Not implemented`). + + **NOTE**: This test suite is in active development. The implementation status based on all other test results. +* ## How to build Run the following command in build directory: @@ -15,21 +19,36 @@ Run the following command in build directory: make conformanceTests ``` -## How to run +## How to generate Conformance operation set +Run the following commands: +1. Clone [`Open Model Zoo repo`](https://github.com/openvinotoolkit/open_model_zoo) +2. Download all possible models using [Downloader tool](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/downloader.py) from + the repo. +3. Convert downloaded models to IR files using [Converter tool](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/converter.py) from the repo. +4. Run [Subgraph dumper](./../subgraphs_dumper/README.md) to collect unique operation set from the models. +> **NOTE**: +> +> The conformance suite run in internal CI checks is based on `2021.4` release. **Please, use `Model optimizer`, 'Subgraph dumper' and `Open Model Zoo` tools +> from `2021.4`.** + +## How to run operation conformance suite The target is able to take the following command-line arguments: * `-h` prints target command-line options with description. * `--device` specifies target device. * `--input_folders` specifies folders with IRs to run. The separator is `,`. * `--plugin_lib_name` is name of plugin library. The example is ov_intel_cpu_plugin. Use only with unregistered in IE Core devices. * `--disable_test_config` allows to ignore all skipped tests with the exception of `DISABLED_` prefix using. -* `--skip_config_path` allows to specify paths to files contain regular expressions list to skip tests. +* `--skip_config_path` allows to specify paths to files contain regular expressions list to skip tests. [Examples](./op_conformance_runner/skip_configs) +* `--config_path` allows to specify path to file contains plugin config. [Example](./op_conformance_runner/config/config_example.txt) * `--extend_report` allows not to re-write device results to the report (add results of this run to the existing). Mutually exclusive with --report_unique_name. * `--report_unique_name` allows to save report with unique name (report_pid_timestamp.xml). Mutually exclusive with --extend_report. * `--save_report_timeout` allows to try to save report in cycle using timeout (in seconds). * `--output_folder` Paths to the output folder to save report. * All `gtest` command-line parameters -The result of execution is `report.xml` file. It demonstrates tests statistic like pass rate, passed, crashed, skipped and failed tests per operation for +The result of execution is `report.xml` file. It demonstrates tests statistic like pass rate, passed, crashed, skipped failed tests and plugin implementation +per +operation for devices. > **NOTE**: @@ -37,6 +56,8 @@ devices. > Using of GTest parallel tool to run `conformanceTests` helps to report crashed tests and collect correct statistic > after unexpected crashes. > +> Use [`Gtest parallel`](https://github.com/google/gtest-parallel) from official repository with [this fix](https://github.com/google/gtest-parallel/pull/76). +> > The example of usage is: > ``` > python3 gtest_parallel.py /path/to/openvino/bin/intel64/Debug/conformanceTests -d . @@ -53,9 +74,15 @@ devices. > python3 merge_xmls.py --input_folders=/path/to/temp_output_report_folder --output_folder=/path/to/output_report_folder --output_filename=report_aggregated > ``` -## How to build operation coverage report +## How to create operation conformance report Run [the script](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/summarize.py) to generate `html` report. The example of using the script is: ``` python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/ -``` \ No newline at end of file +``` +> **NOTE**: +> +> Please, do not forget to copy [styles folder](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/template) to the output directory. It +> helps to provide report with the filters and other usable features. + +Report contains statistic based on conformance results and filter fields at the top of the page. \ No newline at end of file diff --git a/src/tests/functional/plugin/conformance/test_runner/conformance_infra/include/gflag_config.hpp b/src/tests/functional/plugin/conformance/test_runner/conformance_infra/include/gflag_config.hpp index 98a47ba8f46..b8dd12c90b8 100644 --- a/src/tests/functional/plugin/conformance/test_runner/conformance_infra/include/gflag_config.hpp +++ b/src/tests/functional/plugin/conformance/test_runner/conformance_infra/include/gflag_config.hpp @@ -26,7 +26,7 @@ static const char save_report_timeout_message[] = "Optional. Allow to try to sav "Default value is 60 seconds"; static const char skip_config_path_message[] = "Optional. Allows to specify paths to files contain regular expressions list to skip tests. " "Delimiter is `,` symbol. Default value is empty string."; -static const char config_path_message[] = "Optional. Allows to specify path to files contain plugin config. " +static const char config_path_message[] = "Optional. Allows to specify path to file contains plugin config. " "Default value is empty string.";