[GA] Parallel tests (#18773)

* test job

* added script to the test package

* test call fix

* switched test to large runner

* Added option to split tests by suites

* extended logs

* enabled test cache

* fixed workload

* optimized splitting mode

* excluded disabled suites

* temroary removed parallel logs

* added failed logs

* fixed empty name in suites

* test on 4 cores

* make step optional

* fixed param

* test

* grouping suites

* set suite arg

* increase test timeout

* test commit

* test pip deps

* include requirements.txt to the test package

* fixed deps step order

* fixed test counter

* fixed smart filter for suites

* clean up

* disabled repeat failed tests

* review comments

* use runtime execution time for skipped tests

* removed disabled suites

* reduced command lines

* enabled tests results

* fixed typo

* removed unused argument pp

* Log improvements

* merge cached and runtime filters

* fixed order

* fixed init list error

* fixed cache writing

* enable windows pipeline

* changed runner for windows

* optimized balancing using heap

* Fixed test counter

* fixed windows pipeline

* extended logging

* changed pipelines

* added logs on Windows

* fixed pipelines

* debug

* removed os specific code

* fixed "#"

* fixed test results

* fixed win pipeline

* cleanup debug

* rebase fixes

* windows pip requirements

* aligned run_conformance.py

* Apply suggestions from code review

Co-authored-by: Andrey Kashchikhin <andrey.kashchikhin@intel.com>

* reverted windows changes

* reverted build runner

* fixed review comments

* minor review fixes

* make help func static

* renamed test runner

* fixed merge issue

* removed unused log

* reduced command line

* fixed issue fith conformance run

* fixed typo

* set testa  as default split unit

* fixed tasks queue  with time -1

* fixed test result caculation

* reverted wrong fix

* reverted changes

* set time limitation

* reverted unused change

* fix win command lines

* reuse env variables in pipeline

* fixed install files permissions

* fixed pipeline syntax

* reset validation schema

* fixed env names

* reverted initial setting of env

* increased test runner

* fixed pathes

* reuse env path

* reset validation schema

* Revert "reuse env path"

This reverts commit 97422ac595.

* Revert "increased test runner"

This reverts commit 010aa31641.

* revert command line reduction

* made if condition clearer

---------

Co-authored-by: Andrey Kashchikhin <andrey.kashchikhin@intel.com>

.github/workflows/linux.yml

@@ -749,10 +749,12 @@ jobs:
     defaults:
       run:
         shell: bash
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-latest-4-cores
     env:
       INSTALL_DIR: ${{ github.workspace }}/install
       INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests
+      PARALLEL_TEST_SCRIPT: ${{ github.workspace }}/install/tests/functional_test_utils/run_parallel.py
+      PARALLEL_TEST_CACHE: ${{ github.workspace }}/install/tests/test_cache.lst
 
     steps:
       - name: Create Directories
@@ -784,15 +786,34 @@ jobs:
             tar -xzf openvino_tests.tar.gz -C ${{ env.INSTALL_DIR }} && rm openvino_tests.tar.gz || exit 1
           popd
 
-      - name: Intel CPU plugin func tests
+      - name: Install python dependencies
+        run: |
+          python3 -m pip install --upgrade pip
+          python3 -m pip install -r ${{ env.INSTALL_TEST_DIR }}/functional_test_utils/requirements.txt
+
+      - name: Cache Tests Execution Time
+        id: tests-functional-cpu-cache
+        uses: actions/cache@v3
+        with:
+          path: ${{ env.PARALLEL_TEST_CACHE }}
+          key: ${{ runner.os }}-tests-functional-cpu-cache
+
+      - name: Intel CPU plugin func tests (parallel)
         run: |
           source ${{ env.INSTALL_DIR }}/setupvars.sh
-          ${{ env.INSTALL_TEST_DIR }}/ov_cpu_func_tests --gtest_print_time=1 --gtest_filter=*smoke* --gtest_output=xml:"${{ env.INSTALL_TEST_DIR }}/TEST-CPUFuncTests.xml"
+          python3 ${{ env.PARALLEL_TEST_SCRIPT }} -e ${{ env.INSTALL_TEST_DIR }}/ov_cpu_func_tests -c ${{ env.PARALLEL_TEST_CACHE }} -w ${{ env.INSTALL_TEST_DIR }} -s suite -rf 0 -- --gtest_print_time=1 --gtest_filter=*smoke*
+        timeout-minutes: 25
 
       - name: Upload Test Results
         uses: actions/upload-artifact@v3
         if: ${{ always() }}
         with:
           name: test-results-functional-cpu
-          path: ${{ env.INSTALL_TEST_DIR }}/TEST*.xml
+          path: |
+            ${{ env.INSTALL_TEST_DIR }}/TEST*.xml
+            ${{ env.INSTALL_TEST_DIR }}/logs/failed/*.log
+            ${{ env.INSTALL_TEST_DIR }}/logs/crashed/*.log
+            ${{ env.INSTALL_TEST_DIR }}/logs/hanged/*.log
+            ${{ env.INSTALL_TEST_DIR }}/logs/interapted/*.log
+            ${{ env.INSTALL_TEST_DIR }}/logs/disabled_tests.log
           if-no-files-found: 'error'