980904c9ec
* Moved and merged mo/ and extensions/ into openvino/tools/mo
* edited imports
* edited docs to use mo script from entry_point
* edited MO transformations list loading and setup.py
* changed full path -> 'mo' entry point in docs (leftovers)
* corrected package_BOM
* updated resolving --transformation_config in cli_parser.py
* pkgutil-style __init__.py, added summarize_graph into entry points
* updated DOCs for the new --transformations_config
* fix select
* updated install instructions, fixed setup.py for windows and python_version < 3.8
* fixed typo in requirements.txt
* resolved conflicts
* removed creating custom __init__.py from setup.py
* corrected folder with caffe proto
* corrected loading user defined extensions
* fix openvino.tools.mo import in serialize.py
* corrected layer tests for new namespace
* fix in get_testdata.py
* moved model-optimizer into tools/
* renamed import in POT
* corrected mo.yml
* correct CMakeLists.txt for the newest tools/mo
* corrected find_ie_version.py
* docs and openvino-dev setup.py update for the newest tools/mo
* miscellaneous leftovers and fixes
* corrected CI files, pybind11_add_module in CMakeLists.txt and use of tools/mo path instead of tools/model_optimizer
* add_subdirectory pybind11 for tools/mo
* POT path fix
* updated setupvars.sh setupvars.bat
* Revert "updated setupvars.sh setupvars.bat"
This reverts commit c011142340.
* removed model-optimizer env variables from setupvars
* updated CMakeLists.txt to pack MO properly with tests component
* corrected left imports, corrected loading requirements for layer tests
* mo doc typo correction
* minor corrections in docs; removed summarize_graph from entry_points
* get_started_windows.md, MonoDepth_how_to.md corrections, mo path corrections
Stress Tests Suite
This test suite contains tests evaluating the behavior of various OpenVINO use cases under stress conditions:
-
MemCheckTests measuring memory required for the use cases and fail when memory usage exceeds a pre-defined level.
-
StressMemLeaksTests ensure that the use cases does not increase memory levels when executing continuously.
-
StressUnitTests executing various Inference Engine use cases in parallel threads and processes.
Each test refers to configuration files located in <test dir>\.automation
folder.
Getting Started
Stress tests are based on the googletest framework. You can filter tests with
--gtest_filter and explore tests available with --gtest_list_tests options.
Tests measuring memory have a temporary limitation - those need to be executed one at a time to mitigate memory statistics pollution. You can use gtest-parallel for massive tests execution.
Pre-requisites
-
Linux OS to build the tests.
-
gtest-parallel to execute tests.
Building Tests
To build the tests, you need to have OpenVINO™ installed or build from source. Before build the tests, open a terminal, set OpenVINO™ environment, and after that run the commands below:
source <OpenVINO_install_dir>/setupvars.sh
mkdir build && cd build
cmake .. && make -j$(nproc)
Preparing Test Data
Stress tests may work with models from Open Model Zoo. To use it,
download and convert models to IRs using ./scripts/get_testdata.py script.
Script will update test config file with data required for OMZ models execution.
From Intel network you can use models from cache at vdp_tests file share.
Refer to VDP shared folders on using file shares.
Running Tests
gtest-parallel <openvino_bin>/StressMemLeaksTests
For MemCheckTests preferable way is:
python ./scripts/run_memcheck.py --gtest_parallel <gtest_parallel_py_path>
<openvino_bin>/MemCheckTests -- --test_conf=<test_conf_path> --refs_conf=<refs_conf_path>
MemCheckTests logs can be used to gather reference values based on current memory consumption:
mkdir -p MemCheckTests-logs && \
gtest-parallel -d ./MemCheckTests-logs ./MemCheckTests && \
grep -rh ./MemCheckTests-logs -e ".*<model " | sed -e "s/.*<model /<model /" | sort