From e0a75b78d16b612882aa6b35108364c1677e0e14 Mon Sep 17 00:00:00 2001 From: Irina Efode Date: Thu, 24 Aug 2023 18:41:42 +0400 Subject: [PATCH] [CONFORMANCE] Update runner readme (#19393) --- .../plugin/conformance/test_runner/README.md | 94 ++++++++++++------- 1 file changed, 59 insertions(+), 35 deletions(-) diff --git a/src/tests/functional/plugin/conformance/test_runner/README.md b/src/tests/functional/plugin/conformance/test_runner/README.md index 628567d23aa..90735507f73 100644 --- a/src/tests/functional/plugin/conformance/test_runner/README.md +++ b/src/tests/functional/plugin/conformance/test_runner/README.md @@ -12,18 +12,22 @@ The suite checks the following OpenVINO API entities in a plugin implementation: * infer request Also, there are test instantiations to validate hardware plugin functionality via software plugins (for example, MULTI, HETERO, etc.) for the entities. -The other part of the API conformance suite is QueryModel validation: -* `ReadIR_queryModel` tests validate the `query_model` API, using a simple single operation graph (Conformance IR) based on model parameters. -* `OpImplCheck` tests are simple synthetic checks to `query_model` and set implementation status for each operation. +API conformance contains 2 test types: +* `mandatory` scope validates functionality required to be implemented in plugin. The expected pass-rate of mandatory tests is 100% for conformant harware plugin. +* `optional` scope validates both not key functionality in an OpenVINO plugin. OpenVINO provides API, but plugin is able to decide not to implement these methods. Optional scope contains all API Conforformance tests. -A result of the `apiConformanceTests` run is two *xml* files: `report_api.xml` and `report_opset.xml`. The first one shows OpenVINO API entities' test statistics for each OpenVINO API entity, such as `passed/failed/crashed/skipped/hanging`, tests number, pass rates, and implementation status. The second one demonstrates the `query_model` results for each operation. +>**NOTE:** To run only mandatory API Conformance tests use `--gtest_filter=*mandatory*`. + +A result of the `apiConformanceTests` run is report `report_api.xml`. It shows OpenVINO API entities' test statistics for each OpenVINO API entity, such as `passed/failed/crashed/skipped/hanging`, tests number, pass rates, and implementation status. ### Opset Conformance -The suite validates an OpenVINO operation plugin implementation, using simple single operation graphs (Conformance IR) taken from models. The plugin inference output is compared with the reference. +The other part of the API conformance suite is QueryModel validation: The suite contains: -* `ReadIR_compareWithRefs` set allows reading IRs from folders recursively, inferring them, and comparing plugin results with the reference. +* `ReadIR_Inference` set allows reading model based graphs from folders recursively, inferring them, and comparing plugin results with the reference. +* `ReadIR_QueryModel` tests validate the `query_model` API (support the operation by plugin), using a simple ]graph (Conformance IR) extracted from models. +* `ReadIR_ImportExport` tests exports and imports of compiled model, using a graph (Conformance IR) based on model parameters. * `OpImplCheckTest` set checks an operation plugin implementation status, using a simple synthetic single operation graph (`Implemented`/`Not implemented`). The suite checks only `compile_model` without comparison with the reference. A result of the `conformanceTests` run is the `report_opset.xml` file. It shows tests statistic, like pass rate, passed, crashed, skipped, failed tests, and plugin implementation per operation for devices. @@ -52,37 +56,46 @@ There is a simple python runner to complete the whole conformance pipeline local ### The conformance pipeline steps: -1. (Optional) Download models/conformance IR via URL / copy archive to working directory / verify dirs / check list-files. -2. (Optional) Run `SubgraphDumper` to generate a simple single op graph based on models or download the `conformance_ir` folder. (if `-s=1`) +1. (Optional: Applicable only for Opset Conformance suite) Download models/conformance IR via URL / copy archive to working directory / verify dirs / check list-files. +2. (Optional: Applicable only for Opset Conformance suite) Run `SubgraphDumper` to extract graph from models or download the `conformance_ir` folder. (if `-s=1`) 3. Run conformance test executable files. 4. Generate conformance reports. ### Command-line arguments -The script has the following arguments: -* `-h, --help` show this help message and exit -* `-m MODELS_PATH, --models_path MODELS_PATH` - Path to the directory/ies containing models to dump subgraph (the default method is to download conformance IR). It may be a directory, an archive file, an `.lst` file, or a URL to download some data. If `--s=0`, specify the Conformance IRs directory. -* `-d DEVICE, --device DEVICE` - Specify the target device. The default value is `CPU`. -* `-ov OV_PATH, --ov_path OV_PATH` - OV repo path. The default method is to try to find the absolute path of OV repo (by using the script path). -* `-w WORKING_DIR, --working_dir WORKING_DIR` - Specify a working directory to save all artifacts, such as reports, models, `conformance_irs`, etc. -* `-t TYPE, --type TYPE` - Specify conformance type: `OP` or `API`. The default value is `OP`. -* `-s DUMP_CONFORMANCE, --dump_conformance DUMP_CONFORMANCE` - Set `1` if you want to create Conformance IRs from custom/downloaded models. In other cases, set `0`. The default value is `1`. -* `-j WORKERS, --workers WORKERS` - Specify number of workers to run in parallel. The default value is CPU count - `1` -* `--gtest_filter GTEST_FILTER` - Specify gtest filter to apply when running a test. For example, *Add*:*BinaryConv*. The default value is `None`. -* `-c OV_CONFIG_PATH, --ov_config_path OV_CONFIG_PATH` - Specify path to a file, which contains plugin config. -* `-sh SHAPE_MODE, --shape_mode SHAPE_MODE` - Specify shape mode for conformance. The default value is ``. Possible values: `static`, `dynamic`, `` +The script has the following optional arguments: +* `h, --help` Show this help message and exit +* `d DEVICE, --device DEVICE` + Specify a target device. The default value is `CPU` +* `t TYPE, --type TYPE` + Specify conformance type: `OP` or `API`. The default value is `OP` +* `-gtest_filter GTEST_FILTER` + Specify gtest filter to apply for a test run. E.g. *Add*:*BinaryConv*. The default value is `None` +* `w WORKING_DIR, --working_dir WORKING_DIR` + Specify a working directory to save a run artifacts. The default value is `current_dir/temp`. +* `m MODELS_PATH, --models_path MODELS_PATH` + Path to the directory/ies containing models to dump subgraph (the default way is to download conformance IR). It may be directory, archieve file, .lst file with models to download by a link, model file paths. If --s=0, specify the Conformance IRs directory. NOTE: Applicable only for Opset Conformance. +* `ov OV_PATH, --ov_path OV_PATH` + OV binary path. The default way is to find the absolute path of latest bin in the repo (by using script path) +* `j WORKERS, --workers WORKERS` + Specify number of workers to run in parallel. The default value is `CPU_count` +* `c OV_CONFIG_PATH, --ov_config_path OV_CONFIG_PATH` + Specify path to a plugin config file as `.lst` file. Default value is `` +* `s DUMP_GRAPH, --dump_graph DUMP_GRAPH` + Set '1' to create Conformance IRs from models using subgraphsDumper tool. The default value is '0'. + NOTE: Applicable only for Opset Conformance. +* `sm SPECIAL_MODE, --special_mode SPECIAL_MODE` + Specify shape mode (`static`, `dynamic` or ``) for Opset conformance or API scope type (`mandatory` or ``). Default value is `` +* `p PARALLEL_DEVICES, --parallel_devices PARALLEL_DEVICES` + Parallel over HW devices. For example run tests over `GPU.0` and `GPU.1` in case when device are the same +* `f EXPECTED_FAILURES, --expected_failures EXPECTED_FAILURES` + Excepted failures list file path as csv +* `u EXPECTED_FAILURES_UPDATE, --expected_failures_update EXPECTED_FAILURES_UPDATE` + Overwrite expected failures list in case same failures were fixed +* `-cache_path CACHE_PATH` + Path to the cache file with test_name list sorted by execution time as `.lst` file! -> **NOTE**: All arguments are optional and have default values to reproduce OMZ conformance results in a default method. +> **NOTE**: All arguments are optional and have default values to reproduce OMZ based Opset conformance results on `CPU` in a default method. > **NOTE**: The approach can be used as custom model scope validator! @@ -98,7 +111,11 @@ python3 run_conformance.py -m /path/to/new/model_irs -s=1 -w /path/to/working/di ``` 3. Use custom OV build to check GNA conformance, using pre-generated `conformance_irs`: ``` -python3 run_conformance.py -m /path/to/conformance_irs -s=0 -ov /path/to/ov_repo_on_custom_branch -d GNA +python3 run_conformance.py -m /path/to/conformance_irs -s=0 -ov /path/to/ov_repo_on_custom_branch/bin/intel64/Debug -d GNA +``` +4. Use the default method to reproduce opset conformance results for OMZ on TEMPLATE using custom config file: +``` +python3 run_conformance.py -d GPU -c /path/to/ov/config.lst ``` > **IMPORTANT NOTE:** If you need to debug some conformance tests, use the binary run as the default method. If you want to get conformance results or reproduce CI behavior, use the simple python runner. @@ -111,7 +128,11 @@ Run the following commands: 3. Convert downloaded models to IR files, using [Converter tool](https://github.com/openvinotoolkit/open_model_zoo/blob/master/tools/model_tools/converter.py) from the repo. 4. Run [Subgraph dumper](./../subgraphs_dumper/README.md) to collect unique operation set from the models. -## How to run operation conformance suite +> **NOTE:** There are ready to use Conformance IRs based on latest OMZ scope in [the public storage](https://storage.openvinotoolkit.org/test_data/conformance_ir/conformance_ir.tar). + +> **NOTE:** You can use the above algorithm to generate Conformance IRs using a custom model scope supported by OpenVINO. + +## How to run a conformance suite The target is able to take the following command-line arguments: * `-h` prints target command-line options with description. @@ -155,12 +176,15 @@ The target is able to take the following command-line arguments: > python3 merge_xmls.py --input_folders=/path/to/temp_output_report_folder --output_folder=/path/to/output_report_folder --output_filename=report_aggregated > ``` -## How to create operation conformance report +## How to create a conformance report Run [the summarize script](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/summarize.py) to generate `html` and `csv` report. Check command-line arguments with `--help` before running the command. The example of using the script is: ``` -python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/ +python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report_opset.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/ -t OP +``` +``` +python3 summarize.py --xml /opt/repo/infrastructure-master/thirdparty/gtest-parallel/report_api.xml --out /opt/repo/infrastructure-master/thirdparty/gtest-parallel/ -t API ``` > **NOTE**: Remember to copy [styles folder](./../../../../ie_test_utils/functional_test_utils/layer_tests_summary/template) to the output directory. It helps to provide a report with filters and other useful features.