Commit Graph

55 Commits

Author SHA1 Message Date
Vitaliy Urusovskij
4bfb353dbd Add fillBlobs to fix issue with negative values in image_info input layer (#3271) 2020-11-23 17:32:49 +03:00
Vitaliy Urusovskij
9f427bbf6e Add WA for OMZ Keras models in stress tests (#3272) 2020-11-23 17:32:35 +03:00
Vitaliy Urusovskij
dee00cdc92 Add new model to tgl_test_config.yml (#3236) 2020-11-20 13:33:57 +03:00
Vitaliy Urusovskij
b5930eb58e Fix "results" uploading to DB in time_tests (#3215) 2020-11-20 10:49:31 +03:00
Vitaliy Urusovskij
070fc17087 Extend information to submit to a DB in time_tests (#3018)
* Optimize imports in time_tests conftest.py

* Extend information to submit to a DB in time_tests:
1. Add `--manifest` and `--db_metadata` CLI keys
2. Add `prepare_db_info` fixture

* Move `validate_test_case` fixture to a const schema, remove extra checks

* Add `manifest_metadata` fixture to parse and validate manifest only once

* Add OS utils to submit os info to DB
2020-11-17 12:07:40 +03:00
Vitaliy Urusovskij
0ef1fb8a09 Update MemCheckTests OMZ part: get_testdata.py and configs (#2722) 2020-10-30 00:37:02 +03:00
Vitaliy Urusovskij
e1f4585cb8 Add validate_test_case fixture with using of jsonschema. Specify all required fields for test cases (#2821) 2020-10-29 00:11:01 +03:00
Vitaliy Urusovskij
3c5aefb427 Remove memcheck_pregen_irs_tests MemCheck configs due obsolescence (#2693) 2020-10-19 09:48:38 +03:00
Vitaliy Urusovskij
12b690952c Time tests improvements (#2642)
* Remove extra functions from run_timetest.py

* Add `log.debug` of raw and aggregated statistics in run_timetest.py

* Implement storing of models locally for test_timetest.py
2020-10-14 20:05:52 +03:00
Vitaliy Urusovskij
a4fe59ba0a Specify cl_cache_dir env var: (#2581)
1. General behaviour doesn't work for some compute-runtime versions.
Specification of `cl_cache_dir` fixes it
2020-10-08 15:02:31 +03:00
Vitaliy Urusovskij
474dcc277f Prepare new test configuration for test_timetest.py (#2561)
* Add logging of DB upload to tconf with refs saving

* Prepare `.automation/tgl_test_config.yml` with models from VPUX package

* Add restriction on `exe` and `model` existence in run_timetest.py
2020-10-08 10:30:43 +03:00
Vitaliy Urusovskij
c3ff2948a4 Timetests test_runner improvements (#2552)
* Remove `generate_tmp_path` as unnecessary after refactoring

* Add `check_positive_int` check for `-niter` CLI key

* Replace `TestConfDumper` with number of fixtures:
1. Save all test info in global `request` and `pytestconfig` fixtures
2. Add `test_info` fixture for retrieving test info from test
3. Add `prepare_tconf_with_refs` fixture for test conf dump
2020-10-07 17:15:02 +03:00
Vitaliy Urusovskij
6949d93e42 Add "cl_cache" preparation in test_timetest.py (#2490) 2020-10-06 14:16:50 +03:00
Vitaliy Urusovskij
05d70cfa43 Remove --collect_results_only (#2509)
* Remove CLI keys from README

* Remove `--collect_results_only` from MemCheckTests
2020-10-05 17:43:24 +03:00
Vitaliy Urusovskij
639052d0b8 Support references in test_timetest.py (#2431) 2020-10-01 11:35:57 +03:00
Vitaliy Urusovskij
c8530201dd Support DB reporting for test_timetest.py (#2359) 2020-10-01 08:39:19 +03:00
Vitaliy Urusovskij
0bd966ad08 Add fill_inputs step in timetests (#2413) 2020-09-25 14:08:03 +03:00
Vitaliy Urusovskij
d48d2109dc Test timetest MVP (#2345) 2020-09-21 21:33:42 +03:00
Vitaliy Urusovskij
babff1fd15 Extend timetest_infer pipeline with infer track and blob's support (#2298) 2020-09-21 02:09:41 +03:00
Evgeny Talanin
fe99e055f2 Bump cmake version to 3.13 (#2258) 2020-09-18 18:58:12 +03:00
Vitaliy Urusovskij
20266276ec New layout for time_tests (#2321)
* Change folders' layout for `time_tests`

* Refactoring of namings in `time_tests`
2020-09-18 18:53:49 +03:00
Vitaliy Urusovskij
383152db28 Fix TimeTests build and README (#2275)
* Fix searching of pipelines for `time_tests` build

* Add `realpath` use in README for IEDevPackage because of limitation

* Add `mkdir build && cd build` commands in README

* Rename `time-tests` to `time_tests` in README
2020-09-17 13:01:40 +03:00
Andrey Somsikov
e35b720e8f Fix time_tests gcc 4.8.5 build error (#2208) 2020-09-17 11:50:40 +03:00
Andrey Somsikov
f84a6d97ac Split time-tests common library (#2173)
Split time-tests common library

Add a README.md with workflow description.
Defined "timetest_" suffix for all time tests.
Applied clang-format-9 and added a README.md

Co-authored-by: Alina Alborova <alina.alborova@intel.com>
2020-09-14 09:04:49 +03:00
Vitaliy Urusovskij
a8f8ab8c12 Implement run_executable.py to run TimeTests several times (#2125) 2020-09-10 19:41:23 +03:00
Vitaliy Urusovskij
4bd05c5364 Implement statistics collection: (#2056)
1. Add `-s` CLI key to get statistics file path
2. Implement `StatisticsWriter` singleton to manage handle to this file
2020-09-09 15:30:23 +03:00
Andrey Somsikov
b8a8fe7c0d Sort memcheck timeline report for failures (#2088) 2020-09-09 10:10:22 +03:00
Vitaliy Urusovskij
867340e8f1 Add runPipeline wrapper to mandatory track full run (#2100) 2020-09-09 00:13:07 +03:00
Vitaliy Urusovskij
0b61a7568f First time to inference POC (#1964) 2020-09-03 22:08:37 +03:00
Vitaliy Urusovskij
2386c5d376 [Stress] Define --timeout in run_memcheck.py used in gtest-parallel (#1576) 2020-08-05 12:33:01 +03:00
Vitaliy Urusovskij
f8513d8fd3 [Stress] Remove --env_conf at all after deprecation (#1582) 2020-08-03 12:58:27 +03:00
Vitaliy Urusovskij
25cadb661b [Stress] Enable OMZ info_dumper.py in get_testdata.py (#1485)
* [Stress] Support OMZ model_info.py in get_testdata.py

* [Stress] Copy IRs from OMZ models folder to IRs folder

* [Stress] Support modified configs in C++ tests

* [Stress] Deprecate support of --env_conf due refactoring of configs

* [Stress] Update configs:
1. Removed env configs due deprecation
2. Moved test configs to a new format

* [Stress] Extend MemCheck records with info from test config
2020-07-31 12:13:12 +03:00
Andrey Somsikov
9299e32df0 Add product version to memcheck records (#1508) 2020-07-30 10:17:56 +03:00
Vitaliy Urusovskij
6fcad95c71 [Stress] Redesign of MemCheckTests (#650)
* [Stress] Redesigned MemCheckTests: 1. Added MemCheckPipeline to incapsulate measures and logging. 2. Moved references to array

* [Stress] Added tracking of THREADS in MemCheckTests
2020-07-23 11:07:13 +03:00
Vitaliy Urusovskij
5665ec9b26 [Stress] Small improvement of scripts (#1302)
* [Stress] Extend CLI with arguments checks for compare_memcheck_2_runs.py

* [Stress] Extend requirements.txt with PyYAML
2020-07-22 10:24:15 +03:00
Vitaliy Urusovskij
57779511d6 [Stress] Integrate compare_memcheck_2_runs call in run_memcheck.py (#1036)
* [Stress] Define Database constant arguments in memcheck_upload.py only

* [Stress] Simplify computations using HashableDict in `compare_memcheck_2_runs`

* [Stress] Add comparison using pandas
2020-07-20 10:23:51 +03:00
Vitaliy Urusovskij
930d687ed9 [Stress] Specify OMZ develop branch in get_testdata.py (#1318)
Since in validation OMZ is used from a package (and OMZ is from develop branch), align OMZ default for get_testdata.py
2020-07-16 15:28:25 +03:00
Vitaliy Urusovskij
c0b28daf9c Fix MemCheckTests failures caused by change in OMZ models scope (#1214)
Run only models available in OMZ to prevent failures
Remove person-reidentification-retail-0031.xml from configs
2020-07-08 12:33:39 +03:00
Vitaliy Urusovskij
e085a04d8c [Stress] Move logging after all computations of progress_str (#1226) 2020-07-07 14:23:26 +03:00
Vitaliy Urusovskij
a17366f621 [Stress] Enable StressMemleaksTests on GPU in precommit (#1079)
* [Stress] Fix missing retry for StressMemLeaksTests

* [Stress] Add smoothing with sliding average for StressMemleaksTests

* [Stress] Enable GPU in StressMemleaksTests precommit scope
2020-07-03 11:57:16 +03:00
Vitaliy Urusovskij
0e904405f7 [Stress] Add --ref_db_collection in compare_memcheck_2_runs.py (#1157) 2020-07-02 13:14:07 +03:00
Ilya Lavrenov
a7579d5c35 Removed ICNNNetReader interface (#1042)
* Removed ICNNNetReader interface

* Fixed stress tests

* Fixed comments in VPU plugin

* Removed duplicated stress tests

* Fixed watchdog tests
2020-06-23 22:34:26 +03:00
Andrey Somsikov
52a42624e9 Add memcheck runner script (#1031)
Script executes measurement isolated with gtest-parallel,
handles database uploading and reports generation.
2020-06-22 16:23:09 +03:00
Vitaliy Urusovskij
3417004e6d Add memcheck comparison script (#935)
Add compare_memcheck_2_runs.py compares two runs.
Add handling of broken files for `parse_memcheck_log`
2020-06-19 15:56:32 +03:00
Andrey Somsikov
626bc4f3d4 Add commit links to memcheck report (#820) 2020-06-08 15:08:58 +03:00
Vitaliy Urusovskij
72d9a9fae7 Use pre-defined DB collection names in memcheck_upload.py CLI (#651)
Use argparses `choices` for `--db_collection` option.

Also removed unnecessary redefinition of `db_collection` in memcheck_upload.py
2020-06-03 13:54:38 +03:00
Vitaliy Urusovskij
1d179fdb39 Add parallel downloads to stress tests (#678) 2020-06-02 17:24:22 +03:00
Vitaliy Urusovskij
4a44f84dab [Stress] Updated test_configs with new path to OMZ mtcnn models (#602) 2020-05-26 20:11:46 +03:00
Alexey Suhov
deb008a26f publish master branch snapshot, revision 8d31237e2c3f673cbb0f0ba110fc10f5cce1d2bb 2020-05-22 02:23:12 +03:00
Alexey Suhov
f30dcc218c publish master branch snapshot, revision 9df5eb1f84e13a35720a918f88324561222ab114 2020-05-20 01:13:06 +03:00