Commit Graph
66 Commits
Author SHA1 Message Date
Victor Kuznetsov 1217d26f2f rename time test config (#3923) 2021-01-21 12:32:15 +03:00
Vitaliy Urusovskij 1e858830e2 Change json.loads to json.load for --db_metadata (#3871) 2021-01-19 18:33:00 +03:00
Ilya Lavrenov 50f9dd8a6b Enabled CMP0025 as NEW (#3791) 2021-01-11 14:48:27 +03:00
Vitaliy Urusovskij 08f54dbac8 Fix metadata_from_manifest() according to new manifest version (#3771) 2020-12-31 09:57:58 +03:00
Vitaliy Urusovskij fe229ba19d Move time_tests to new manifest version (#3759) 2020-12-30 17:28:22 +03:00
Ilya Lavrenov 278b662e56 Removed global CMAKE_MODULE_PATH in IE DevPackage (#3747)
* Cleanup

* Added CMAKE_MODULE_LINKER_FLAGS for onecoreuap toolchain

* Fixed ENABLE_FASTER_BUILD mode on Android

* Cosmetic fix

* Fixed fuzzing

* Enabled fuzzing
2020-12-27 00:02:50 +03:00
Ilya Lavrenov 9465073f58 Introduce IEDevScripts package (#3661)
* Refactored developer package

* Added fuzzing for CMAKE_MODULE_LINKER_FLAGS as well

* Added options for developer package

* More improvements

* Further improvements

* Removed global CMAKE_MODULE_PATH population

* Fixes

* Final fixes

* Fixed python build

* Fix for TBB

* Fixed Find TBB

* Fixed install

* Fixes for OV features

* Split developer targets per component

* Fixed IE build tree config

* Fixed ITT

* Fixed review comments

* Clean export dependencies

* Fixed export of pugixml

* Added IEDevScripts_DIR for Android

* Fixed Android #2

* Fixed Android #3

* Fixed python cc

* Disabled Core threading tests on GNA
2020-12-22 18:44:59 +03:00
Andrey Somsikov 7fe21dc6ee Fix windows build for gflags for time_tests (#3504)
GFlags builds multiple targets, require aligning build
options on windows builds.

FetchModule offloads project configuration to cmake. This also allows
to align build configurations and targets across projects:
https://crascit.com/2015/07/25/cmake-gtest/
2020-12-11 09:25:14 +03:00
Andrey Somsikov 5cc08367b2 Build time_tests with OpenVINO install (#3484) 2020-12-07 12:56:37 +03:00
Vitaliy Urusovskij e910902c9d Fix paths for squeezenet1.1 in time_tests config (#3415) 2020-11-30 18:45:08 +03:00
Vitaliy Urusovskij 364bf08920 Add several new models to tgl_test_config.yml in time_tests (#3268)
* Fix wrong path for `yolo-v2-tiny-ava-0001` for time_tests

* Add several new models to `tgl_test_config.yml` in time_tests
2020-11-24 13:26:16 +03:00
Vitaliy Urusovskij 4bfb353dbd Add fillBlobs to fix issue with negative values in image_info input layer (#3271) 2020-11-23 17:32:49 +03:00
Vitaliy Urusovskij 9f427bbf6e Add WA for OMZ Keras models in stress tests (#3272) 2020-11-23 17:32:35 +03:00
Vitaliy Urusovskij dee00cdc92 Add new model to tgl_test_config.yml (#3236) 2020-11-20 13:33:57 +03:00
Vitaliy Urusovskij b5930eb58e Fix "results" uploading to DB in time_tests (#3215) 2020-11-20 10:49:31 +03:00
Vitaliy Urusovskij 070fc17087 Extend information to submit to a DB in time_tests (#3018)
* Optimize imports in time_tests conftest.py

* Extend information to submit to a DB in time_tests:
1. Add `--manifest` and `--db_metadata` CLI keys
2. Add `prepare_db_info` fixture

* Move `validate_test_case` fixture to a const schema, remove extra checks

* Add `manifest_metadata` fixture to parse and validate manifest only once

* Add OS utils to submit os info to DB
2020-11-17 12:07:40 +03:00
Vitaliy Urusovskij 0ef1fb8a09 Update MemCheckTests OMZ part: get_testdata.py and configs (#2722) 2020-10-30 00:37:02 +03:00
Vitaliy Urusovskij e1f4585cb8 Add validate_test_case fixture with using of jsonschema. Specify all required fields for test cases (#2821) 2020-10-29 00:11:01 +03:00
Vitaliy Urusovskij 3c5aefb427 Remove memcheck_pregen_irs_tests MemCheck configs due obsolescence (#2693) 2020-10-19 09:48:38 +03:00
Vitaliy Urusovskij 12b690952c Time tests improvements (#2642)
* Remove extra functions from run_timetest.py

* Add `log.debug` of raw and aggregated statistics in run_timetest.py

* Implement storing of models locally for test_timetest.py
2020-10-14 20:05:52 +03:00
Vitaliy Urusovskij a4fe59ba0a Specify cl_cache_dir env var: (#2581)
1. General behaviour doesn't work for some compute-runtime versions.
Specification of `cl_cache_dir` fixes it
2020-10-08 15:02:31 +03:00
Vitaliy Urusovskij 474dcc277f Prepare new test configuration for test_timetest.py (#2561)
* Add logging of DB upload to tconf with refs saving

* Prepare `.automation/tgl_test_config.yml` with models from VPUX package

* Add restriction on `exe` and `model` existence in run_timetest.py
2020-10-08 10:30:43 +03:00
Vitaliy Urusovskij c3ff2948a4 Timetests test_runner improvements (#2552)
* Remove `generate_tmp_path` as unnecessary after refactoring

* Add `check_positive_int` check for `-niter` CLI key

* Replace `TestConfDumper` with number of fixtures:
1. Save all test info in global `request` and `pytestconfig` fixtures
2. Add `test_info` fixture for retrieving test info from test
3. Add `prepare_tconf_with_refs` fixture for test conf dump
2020-10-07 17:15:02 +03:00
Vitaliy Urusovskij 6949d93e42 Add "cl_cache" preparation in test_timetest.py (#2490) 2020-10-06 14:16:50 +03:00
Vitaliy Urusovskij 05d70cfa43 Remove --collect_results_only (#2509)
* Remove CLI keys from README

* Remove `--collect_results_only` from MemCheckTests
2020-10-05 17:43:24 +03:00
Vitaliy Urusovskij 639052d0b8 Support references in test_timetest.py (#2431) 2020-10-01 11:35:57 +03:00
Vitaliy Urusovskij c8530201dd Support DB reporting for test_timetest.py (#2359) 2020-10-01 08:39:19 +03:00
Vitaliy Urusovskij 0bd966ad08 Add fill_inputs step in timetests (#2413) 2020-09-25 14:08:03 +03:00
Vitaliy Urusovskij d48d2109dc Test timetest MVP (#2345) 2020-09-21 21:33:42 +03:00
Vitaliy Urusovskij babff1fd15 Extend timetest_infer pipeline with infer track and blob's support (#2298) 2020-09-21 02:09:41 +03:00
Evgeny Talanin fe99e055f2 Bump cmake version to 3.13 (#2258) 2020-09-18 18:58:12 +03:00
Vitaliy Urusovskij 20266276ec New layout for time_tests (#2321)
* Change folders' layout for `time_tests`

* Refactoring of namings in `time_tests`
2020-09-18 18:53:49 +03:00
Vitaliy Urusovskij 383152db28 Fix TimeTests build and README (#2275)
* Fix searching of pipelines for `time_tests` build

* Add `realpath` use in README for IEDevPackage because of limitation

* Add `mkdir build && cd build` commands in README

* Rename `time-tests` to `time_tests` in README
2020-09-17 13:01:40 +03:00
Andrey Somsikov e35b720e8f Fix time_tests gcc 4.8.5 build error (#2208) 2020-09-17 11:50:40 +03:00
Andrey SomsikovandAlina Alborova f84a6d97ac Split time-tests common library (#2173)
Split time-tests common library

Add a README.md with workflow description.
Defined "timetest_" suffix for all time tests.
Applied clang-format-9 and added a README.md

Co-authored-by: Alina Alborova <alina.alborova@intel.com>
2020-09-14 09:04:49 +03:00
Vitaliy Urusovskij a8f8ab8c12 Implement run_executable.py to run TimeTests several times (#2125) 2020-09-10 19:41:23 +03:00
Vitaliy Urusovskij 4bd05c5364 Implement statistics collection: (#2056)
1. Add `-s` CLI key to get statistics file path
2. Implement `StatisticsWriter` singleton to manage handle to this file
2020-09-09 15:30:23 +03:00
Andrey Somsikov b8a8fe7c0d Sort memcheck timeline report for failures (#2088) 2020-09-09 10:10:22 +03:00
Vitaliy Urusovskij 867340e8f1 Add runPipeline wrapper to mandatory track full run (#2100) 2020-09-09 00:13:07 +03:00
Vitaliy Urusovskij 0b61a7568f First time to inference POC (#1964) 2020-09-03 22:08:37 +03:00
Vitaliy Urusovskij 2386c5d376 [Stress] Define --timeout in run_memcheck.py used in gtest-parallel (#1576) 2020-08-05 12:33:01 +03:00
Vitaliy Urusovskij f8513d8fd3 [Stress] Remove --env_conf at all after deprecation (#1582) 2020-08-03 12:58:27 +03:00
Vitaliy Urusovskij 25cadb661b [Stress] Enable OMZ info_dumper.py in get_testdata.py (#1485)
* [Stress] Support OMZ model_info.py in get_testdata.py

* [Stress] Copy IRs from OMZ models folder to IRs folder

* [Stress] Support modified configs in C++ tests

* [Stress] Deprecate support of --env_conf due refactoring of configs

* [Stress] Update configs:
1. Removed env configs due deprecation
2. Moved test configs to a new format

* [Stress] Extend MemCheck records with info from test config
2020-07-31 12:13:12 +03:00
Andrey Somsikov 9299e32df0 Add product version to memcheck records (#1508) 2020-07-30 10:17:56 +03:00
Vitaliy Urusovskij 6fcad95c71 [Stress] Redesign of MemCheckTests (#650)
* [Stress] Redesigned MemCheckTests: 1. Added MemCheckPipeline to incapsulate measures and logging. 2. Moved references to array

* [Stress] Added tracking of THREADS in MemCheckTests
2020-07-23 11:07:13 +03:00
Vitaliy Urusovskij 5665ec9b26 [Stress] Small improvement of scripts (#1302)
* [Stress] Extend CLI with arguments checks for compare_memcheck_2_runs.py

* [Stress] Extend requirements.txt with PyYAML
2020-07-22 10:24:15 +03:00
Vitaliy Urusovskij 57779511d6 [Stress] Integrate compare_memcheck_2_runs call in run_memcheck.py (#1036)
* [Stress] Define Database constant arguments in memcheck_upload.py only

* [Stress] Simplify computations using HashableDict in `compare_memcheck_2_runs`

* [Stress] Add comparison using pandas
2020-07-20 10:23:51 +03:00
Vitaliy Urusovskij 930d687ed9 [Stress] Specify OMZ develop branch in get_testdata.py (#1318)
Since in validation OMZ is used from a package (and OMZ is from develop branch), align OMZ default for get_testdata.py
2020-07-16 15:28:25 +03:00
Vitaliy Urusovskij c0b28daf9c Fix MemCheckTests failures caused by change in OMZ models scope (#1214)
Run only models available in OMZ to prevent failures
Remove person-reidentification-retail-0031.xml from configs
2020-07-08 12:33:39 +03:00
Vitaliy Urusovskij e085a04d8c [Stress] Move logging after all computations of progress_str (#1226) 2020-07-07 14:23:26 +03:00