591c3e61c5
* Partial progress * Finish v1 * Cleanup * Remove useless files * Fix path to pdpd * Fix onnx path * Minor change * Rework MO * Minor change * Remove some costraints * Add MO constraints * Update gitignore for MO * Minor change * Apply tech sync discussion * Cleanup * CR comment * Debug ONNX FE * simplify ONNX FE * Update cmake * Hardcode ONNX requirement * Add dependency resolver to cmake * Add constraints for openvino/tests * Add missing pytest-html * Fix -c path * Revert debug changes to path * Add cmake to copy constraints.txt * Update dependabot * Remove slash * Remove cmake * Debug prints * Minor changes * Move reqs check to separate file * Add requirements parser to benchmark_tool * Fix smoke tests constraints * Minor fixes * Minor change * My fixes were apparently wrong * Debug - self.executable_path * Debug - add singledispatch to tests and tools * Debug - print IE_APP_PATHs * Revert "Debug - print IE_APP_PATHs" This reverts commit67ccb6d3f5. * Revert "Debug - add singledispatch to tests and tools" This reverts commit3b945931e2. * Revert "Debug - self.executable_path" This reverts commit3aa724eff6. * update dependabot * update dependabot * Skip benchmark_app tests * Use CMAKE_CURRENT_BINARY_DIR in cmake * Remove debug prints * minor change --------- Signed-off-by: p-wysocki <przemyslaw.wysocki@intel.com>
Stress Tests Suite
This test suite contains tests evaluating the behavior of various OpenVINO use cases under stress conditions:
-
MemCheckTests measuring memory required for the use cases and fail when memory usage exceeds a pre-defined level.
-
StressMemLeaksTests ensure that the use cases does not increase memory levels when executing continuously.
-
StressUnitTests executing various Inference Engine use cases in parallel threads and processes.
Each test refers to configuration files located in <test dir>\.automation
folder.
Getting Started
Stress tests are based on the googletest framework. You can filter tests with
--gtest_filter and explore tests available with --gtest_list_tests options.
Tests measuring memory have a temporary limitation - those need to be executed one at a time to mitigate memory statistics pollution. You can use gtest-parallel for massive tests execution.
Pre-requisites
-
Linux OS to build the tests.
-
gtest-parallel to execute tests.
Building Tests
To build the tests, you need to have OpenVINO™ installed or build from source. Before build the tests, open a terminal, set OpenVINO™ environment, and after that run the commands below:
source <OpenVINO_install_dir>/setupvars.sh
mkdir build && cd build
cmake .. && cmake --build . -j8
Preparing Test Data
Stress tests may work with models from Open Model Zoo. To use it,
download and convert models to IRs using ./scripts/get_testdata.py script.
Script will update test config file with data required for OMZ models execution.
From Intel network you can use models from cache at vdp_tests file share.
Refer to VDP shared folders on using file shares.
Running Tests
gtest-parallel <openvino_bin>/StressMemLeaksTests
For MemCheckTests preferable way is:
python ./scripts/run_memcheck.py --gtest_parallel <gtest_parallel_py_path>
<openvino_bin>/MemCheckTests -- --test_conf=<test_conf_path> --refs_conf=<refs_conf_path>
MemCheckTests logs can be used to gather reference values based on current memory consumption:
mkdir -p MemCheckTests-logs && \
gtest-parallel -d ./MemCheckTests-logs ./MemCheckTests && \
grep -rh ./MemCheckTests-logs -e ".*<model " | sed -e "s/.*<model /<model /" | sort