[CPU][TESTS] Add options to manage test targets and scope (#14576)

- by using ENABLE_CPU_SUBSET_TESTS_PATH cmake variable one can specify list of relative paths to functional test which will be included into target ov_cpu_func_tests_subset Target was renamed from cpuDebugFuncTests to ov_cpu_func_tests_subset - by using ENABLE_CPU_SPECIFIC_TARGET_PER_TEST=ON one can trigger generating specific target for each test file, i.e. - ov_cpu_func_slt_convolution - ov_cpu_func_subgraph_mha
2023-03-01 08:39:53 +01:00 · 2023-03-01 08:39:53 +01:00 · 051a1c661e
commit 051a1c661e
parent 7f83c2c72d
6 changed files with 163 additions and 60 deletions
--- a/src/plugins/intel_cpu/src/docs/README.md
+++ b/src/plugins/intel_cpu/src/docs/README.md
@ -1,4 +1,10 @@
-# Debug capabilities
+# OpenVINO CPU plugin
+Development documentation of OpenVINO CPU plugin
+
+## Compilation options
+See [Compilation options](compilation_options.md)
+
+## Debug capabilities
 Use the following cmake option to enable debug capabilities:

 `-DENABLE_DEBUG_CAPS=ON`
@ -7,22 +13,7 @@ Use the following cmake option to enable debug capabilities:
 * [Blob dumping](blob_dumping.md)
 * [Graph serialization](graph_serialization.md)
 * [Graph transformation disabling](feature_disabling.md#graph-transformations)
-
-## Debug log
-
-Debug logs starting with `[ DEBUG ]` will be shown after this option is set to ON, and
-each log has prefix in format `source_file_name:line_num function()` indicating the position of the log in source code.
-
-Environment variable `OV_CPU_DEBUG_LOG` controls which debug logs to output by combining
-patterns, typical examples of usages are:
-   - not define it: no debug logs will be output
-   - `-` : all debug logs will be output
-   - `graph.cpp:798;InitEdges` :  only debug logs from "graph.cpp:798" and function "InitEdges" are output
-   - `-graph.cpp:798;InitEdges` :  only debug logs from specified places are not output
-
-Environment variable `OV_CPU_DEBUG_LOG_BRK` can be set to some keywords or a full log line seen previously, if any debug log match with the content in this variable, an `int3` instruction will be executed to trigger breakpoint trap if it's running inside a debugger.
-
-## Performance summary
-set `OV_CPU_SUMMARY_PERF` environment variable to display performance summary at the time when model is being destructed.
-
-Internal performance counter will be enabled automatically. 
+* [Logging](logging.md)
+* Performance summary
+    * set `OV_CPU_SUMMARY_PERF` environment variable to display performance summary at the time when model is being destructed.
+    * Internal performance counter will be enabled automatically. 
--- a/src/plugins/intel_cpu/src/docs/compilation_options.md
+++ b/src/plugins/intel_cpu/src/docs/compilation_options.md
@ -0,0 +1,17 @@
+# Compilation options
+The following compilation options can be helpful in developing CPU plugin
+
+* `ENABLE_DEBUG_CAPS=ON`
+  See [Debug capabilities](README.md#debug_capabilities)
+* `ENABLE_CPU_SUBSET_TESTS_PATH="relative/path/to/test/file"`
+    * Example: `-DENABLE_CPU_SUBSET_TESTS_PATH="single_layer_tests/convolution.cpp subgraph_tests/src/mha.cpp"`
+    * Specifies the list of relative paths to functional tests which will be included into the new target `ov_cpu_func_tests_subset`
+    * When the option is specified, the target `ov_cpu_func_tests` is excluded from the target `all`
+    * Motivation: To reduce overhead of running / debugging the full test scope
+* `DENABLE_CPU_SPECIFIC_TARGET_PER_TEST=ON`
+    * Generates specific make target for every file from `single_layer_tests` and `subgraph_tests` directories
+    * Examples of the generated targets:
+        - `ov_cpu_func_slt_convolution` (single_layer_tests/convolution.cpp)
+        - `ov_cpu_func_subgraph_mha` (subgraph_tests/src/mha.cpp)
+    * All the generated targets are excluded from default target `all`
+    * Motivation: To reduce overhead of running / debugging the full test scope
--- a/src/plugins/intel_cpu/src/docs/logging.md
+++ b/src/plugins/intel_cpu/src/docs/logging.md
@ -0,0 +1,13 @@
+## Debug log
+
+Debug logs starting with `[ DEBUG ]` will be shown after this option is set to ON, and
+each log has prefix in format `source_file_name:line_num function()` indicating the position of the log in source code.
+
+Environment variable `OV_CPU_DEBUG_LOG` controls which debug logs to output by combining
+patterns, typical examples of usages are:
+   - not define it: no debug logs will be output
+   - `-` : all debug logs will be output
+   - `graph.cpp:798;InitEdges` :  only debug logs from "graph.cpp:798" and function "InitEdges" are output
+   - `-graph.cpp:798;InitEdges` :  only debug logs from specified places are not output
+
+Environment variable `OV_CPU_DEBUG_LOG_BRK` can be set to some keywords or a full log line seen previously, if any debug log match with the content in this variable, an `int3` instruction will be executed to trigger breakpoint trap if it's running inside a debugger.
--- a/src/plugins/intel_cpu/tests/functional/CMakeLists.txt
+++ b/src/plugins/intel_cpu/tests/functional/CMakeLists.txt
@ -4,12 +4,6 @@

 set(TARGET_NAME ov_cpu_func_tests)

-# ov_cpu_func_tests is too big for debugging purpose, cpuDebugFuncTests
-# is a specific version for debugging purpose, just set DEBUG_SRC_PATH
-# to the test case to be debugged and debug using cpuDebugFuncTests
-set(DEBUG_TARGET_NAME cpuDebugFuncTests)
-set(DEBUG_SRC_PATH ${CMAKE_CURRENT_SOURCE_DIR}/subgraph_tests/src/conv_sum_broadcast.cpp)
-
 add_library(cpuSpecificRtInfo STATIC $<TARGET_PROPERTY:openvino_intel_cpu_plugin,SOURCE_DIR>/src/utils/rt_info/memory_formats_attribute.hpp
                                     $<TARGET_PROPERTY:openvino_intel_cpu_plugin,SOURCE_DIR>/src/utils/rt_info/memory_formats_attribute.cpp)
 target_link_libraries(cpuSpecificRtInfo PRIVATE openvino::runtime)
@ -33,40 +27,16 @@ addIeTargetTest(
        LINK_LIBRARIES ${LINK_LIBRARIES}
        ADD_CPPLINT
        LABELS
-            CPU
-)
-
-# remove all non-common files from debug
-set(EXCLUDED_SOURCE_PATHS_FOR_DEBUG
-    ${CMAKE_CURRENT_SOURCE_DIR}/behavior
-    ${CMAKE_CURRENT_SOURCE_DIR}/bfloat16
-    ${CMAKE_CURRENT_SOURCE_DIR}/blob
-    ${CMAKE_CURRENT_SOURCE_DIR}/extension
-    ${CMAKE_CURRENT_SOURCE_DIR}/onnx
-    ${CMAKE_CURRENT_SOURCE_DIR}/single_layer_tests
-    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances
-    ${CMAKE_CURRENT_SOURCE_DIR}/subgraph_tests/src)
-
-# add the source file to debug
-set(OBJECT_FILES_FOR_DEBUG
-    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/core_config.cpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/skip_tests_config.cpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/set_device_name.cpp
-    ${DEBUG_SRC_PATH})
-
-addIeTargetTest(
-        NAME ${DEBUG_TARGET_NAME}
-        ROOT ${CMAKE_CURRENT_SOURCE_DIR}
-        INCLUDES ${INCLUDES}
-        EXCLUDED_SOURCE_PATHS ${EXCLUDED_SOURCE_PATHS_FOR_DEBUG}
-        OBJECT_FILES ${OBJECT_FILES_FOR_DEBUG}
-        DEFINES ${DEFINES}
-        DEPENDENCIES ${DEPENDENCIES}
-        LINK_LIBRARIES ${LINK_LIBRARIES}
-        ADD_CPPLINT
-        LABELS
-            CPU
+        CPU
 )

 set_ie_threading_interface_for(${TARGET_NAME})
-set_ie_threading_interface_for(${DEBUG_TARGET_NAME})
+
+include(specific_tests.cmake)
+include(target_per_test.cmake)
+
+# exclude 'ov_cpu_func_tests' from target 'all' if specific tests path was specified
+if(DEFINED ENABLE_CPU_SPECIFIC_TESTS_PATH)
+  set_target_properties(${TARGET_NAME} PROPERTIES
+    EXCLUDE_FROM_ALL ON)
+endif()
--- a/src/plugins/intel_cpu/tests/functional/specific_tests.cmake
+++ b/src/plugins/intel_cpu/tests/functional/specific_tests.cmake
@ -0,0 +1,55 @@
+# Copyright (C) 2018-2023 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+#
+
+# to limit test scope to just a single test file
+# improves debugging expirience
+  # relative path to specifc test file
+if(DEFINED ENABLE_CPU_SUBSET_TESTS_PATH)
+  set(SUBSET_TARGET_NAME ov_cpu_func_tests_subset)
+
+  set(CPU_SUBSET_TEST_ABS_PATH)
+  set(CPU_SUBSET_TEST_DIR)
+
+  foreach(TEST_PATH ${ENABLE_CPU_SUBSET_TESTS_PATH})
+    list(APPEND CPU_SUBSET_TEST_ABS_PATH ${CMAKE_CURRENT_SOURCE_DIR}/${TEST_PATH})
+    get_filename_component(TEST_DIR ${CPU_SUBSET_TEST_ABS_PATH} DIRECTORY)
+    list(APPEND CPU_SUBSET_TEST_DIR ${TEST_DIR})
+  endforeach()
+
+  # exclude every other test file
+  set(EXCLUDED_SOURCE_PATHS_FOR_SUBSET_TEST
+    ${CMAKE_CURRENT_SOURCE_DIR}/behavior
+    ${CMAKE_CURRENT_SOURCE_DIR}/bfloat16
+    ${CMAKE_CURRENT_SOURCE_DIR}/blob
+    ${CMAKE_CURRENT_SOURCE_DIR}/extension
+    ${CMAKE_CURRENT_SOURCE_DIR}/onnx
+    ${CMAKE_CURRENT_SOURCE_DIR}/single_layer_tests
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances
+    ${CMAKE_CURRENT_SOURCE_DIR}/subgraph_tests/src)
+
+  # list of object files required for each test
+  set(REQUIRED_OBJECT_FILES
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/core_config.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/skip_tests_config.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/set_device_name.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/test_utils/cpu_test_utils.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/test_utils/fusing_test_utils.cpp
+    ${CPU_SUBSET_TEST_ABS_PATH})
+
+  addIeTargetTest(
+    NAME ${SUBSET_TARGET_NAME}
+    ROOT ${CMAKE_CURRENT_SOURCE_DIR}
+    INCLUDES ${INCLUDES}
+    EXCLUDED_SOURCE_PATHS ${EXCLUDED_SOURCE_PATHS_FOR_SUBSET_TEST}
+    OBJECT_FILES ${REQUIRED_OBJECT_FILES}
+    DEFINES ${DEFINES}
+    DEPENDENCIES ${DEPENDENCIES}
+    LINK_LIBRARIES ${LINK_LIBRARIES}
+    ADD_CPPLINT
+    LABELS
+    CPU
+  )
+
+  set_ie_threading_interface_for(${SUBSET_TARGET_NAME})
+endif()
--- a/src/plugins/intel_cpu/tests/functional/target_per_test.cmake
+++ b/src/plugins/intel_cpu/tests/functional/target_per_test.cmake
@ -0,0 +1,57 @@
+# Copyright (C) 2018-2023 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+#
+
+# create targed with prefix TARGET_PREFIX for each test file in directory TEST_DIR
+function(create_target_per_test_for_directory TEST_DIR TARGET_PREFIX)
+  file(GLOB LIST_OF_TEST_FILES ${TEST_DIR}/*.cpp)
+
+  # exclude every other test file inside directory
+  set(EXCLUDED_SOURCE_PATHS_FOR_TEST
+    ${TEST_DIR})
+
+  # list of object files required for each test
+  set(REQUIRED_OBJECT_FILES
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/core_config.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/skip_tests_config.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/shared_tests_instances/set_device_name.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/test_utils/cpu_test_utils.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/test_utils/fusing_test_utils.cpp
+  )
+
+  # create targed for each test file in directory
+  foreach(TEST_FILE ${LIST_OF_TEST_FILES})
+    # test file name without extension
+    get_filename_component(TEST_FILE_WE ${TEST_FILE} NAME_WE)
+    set(TEST_TARGET_NAME ${TARGET_PREFIX}_${TEST_FILE_WE})
+
+    # create target
+    addIeTargetTest(
+      NAME ${TEST_TARGET_NAME}
+      ROOT ${TEST_DIR}
+      INCLUDES ${INCLUDES}
+      EXCLUDED_SOURCE_PATHS ${EXCLUDED_SOURCE_PATHS_FOR_TEST}
+      OBJECT_FILES ${REQUIRED_OBJECT_FILES} ${TEST_FILE}
+      DEFINES ${DEFINES}
+      DEPENDENCIES ${DEPENDENCIES}
+      LINK_LIBRARIES ${LINK_LIBRARIES}
+      ADD_CPPLINT
+      LABELS
+      CPU
+    )
+
+    set_ie_threading_interface_for(${TEST_TARGET_NAME})
+    # avoid building binaries for every test in case target 'all' is used
+    set_target_properties(${TEST_TARGET_NAME} PROPERTIES
+      EXCLUDE_FROM_ALL ON)
+  endforeach()
+endfunction()
+
+if(ENABLE_CPU_SPECIFIC_TARGET_PER_TEST)
+  create_target_per_test_for_directory(${CMAKE_CURRENT_SOURCE_DIR}/subgraph_tests/src ov_cpu_func_subgraph)
+  create_target_per_test_for_directory(${CMAKE_CURRENT_SOURCE_DIR}/single_layer_tests ov_cpu_func_slt)
+endif()
+
+# examples of targets:
+# - ov_cpu_func_subgraph_mha
+# - ov_cpu_func_slt_convolution