Feature/azaytsev/from 2021 4 (#9247)

* Added info on DockerHub CI Framework * Feature/azaytsev/change layout (#3295) * Changes according to feedback comments * Replaced @ref's with html links * Fixed links, added a title page for installing from repos and images, fixed formatting issues * Added links * minor fix * Added DL Streamer to the list of components installed by default * Link fixes * Link fixes * ovms doc fix (#2988) * added OpenVINO Model Server * ovms doc fixes Co-authored-by: Trawinski, Dariusz <dariusz.trawinski@intel.com> * Updated openvino_docs.xml * Updated the link to software license agreements * Revert "Updated the link to software license agreements" This reverts commit 706dac500e. * Docs to Sphinx (#8151) * docs to sphinx * Update GPU.md * Update CPU.md * Update AUTO.md * Update performance_int8_vs_fp32.md * update * update md * updates * disable doc ci * disable ci * fix index.rst Co-authored-by: Andrey Zaytsev <andrey.zaytsev@intel.com> # Conflicts: # .gitignore # docs/CMakeLists.txt # docs/IE_DG/Deep_Learning_Inference_Engine_DevGuide.md # docs/IE_DG/Extensibility_DG/Custom_ONNX_Ops.md # docs/IE_DG/Extensibility_DG/VPU_Kernel.md # docs/IE_DG/InferenceEngine_QueryAPI.md # docs/IE_DG/Int8Inference.md # docs/IE_DG/Integrate_with_customer_application_new_API.md # docs/IE_DG/Model_caching_overview.md # docs/IE_DG/supported_plugins/GPU_RemoteBlob_API.md # docs/IE_DG/supported_plugins/HETERO.md # docs/IE_DG/supported_plugins/MULTI.md # docs/MO_DG/prepare_model/convert_model/Convert_Model_From_Caffe.md # docs/MO_DG/prepare_model/convert_model/Convert_Model_From_Kaldi.md # docs/MO_DG/prepare_model/convert_model/Convert_Model_From_MxNet.md # docs/MO_DG/prepare_model/convert_model/Convert_Model_From_ONNX.md # docs/MO_DG/prepare_model/convert_model/Converting_Model.md # docs/MO_DG/prepare_model/convert_model/Converting_Model_General.md # docs/MO_DG/prepare_model/convert_model/Cutting_Model.md # docs/MO_DG/prepare_model/convert_model/pytorch_specific/Convert_RNNT.md # docs/MO_DG/prepare_model/convert_model/tf_specific/Convert_EfficientDet_Models.md # docs/MO_DG/prepare_model/convert_model/tf_specific/Convert_WideAndDeep_Family_Models.md # docs/MO_DG/prepare_model/convert_model/tf_specific/Convert_YOLO_From_Tensorflow.md # docs/doxygen/Doxyfile.config # docs/doxygen/ie_docs.xml # docs/doxygen/ie_plugin_api.config # docs/doxygen/ngraph_cpp_api.config # docs/doxygen/openvino_docs.xml # docs/get_started/get_started_macos.md # docs/get_started/get_started_raspbian.md # docs/get_started/get_started_windows.md # docs/img/cpu_int8_flow.png # docs/index.md # docs/install_guides/VisionAcceleratorFPGA_Configure.md # docs/install_guides/VisionAcceleratorFPGA_Configure_Windows.md # docs/install_guides/deployment-manager-tool.md # docs/install_guides/installing-openvino-linux.md # docs/install_guides/installing-openvino-macos.md # docs/install_guides/installing-openvino-windows.md # docs/optimization_guide/dldt_optimization_guide.md # inference-engine/ie_bridges/c/include/c_api/ie_c_api.h # inference-engine/ie_bridges/python/docs/api_overview.md # inference-engine/ie_bridges/python/sample/ngraph_function_creation_sample/README.md # inference-engine/ie_bridges/python/sample/speech_sample/README.md # inference-engine/ie_bridges/python/src/openvino/inference_engine/ie_api.pyx # inference-engine/include/ie_api.h # inference-engine/include/ie_core.hpp # inference-engine/include/ie_version.hpp # inference-engine/samples/benchmark_app/README.md # inference-engine/samples/speech_sample/README.md # inference-engine/src/plugin_api/exec_graph_info.hpp # inference-engine/src/plugin_api/file_utils.h # inference-engine/src/transformations/include/transformations_visibility.hpp # inference-engine/tools/benchmark_tool/README.md # ngraph/core/include/ngraph/ngraph.hpp # ngraph/frontend/onnx_common/include/onnx_common/parser.hpp # ngraph/python/src/ngraph/utils/node_factory.py # openvino/itt/include/openvino/itt.hpp # thirdparty/ade # tools/benchmark/README.md * Cherry-picked remove font-family (#8211) * Cherry-picked: Update get_started_scripts.md (#8338) * doc updates (#8268) * Various doc changes * theme changes * remove font-family (#8211) * fix css * Update uninstalling-openvino.md * fix css * fix * Fixes for Installation Guides Co-authored-by: Andrey Zaytsev <andrey.zaytsev@intel.com> Co-authored-by: kblaszczak-intel <karol.blaszczak@intel.com> # Conflicts: # docs/IE_DG/Bfloat16Inference.md # docs/IE_DG/InferenceEngine_QueryAPI.md # docs/IE_DG/OnnxImporterTutorial.md # docs/IE_DG/supported_plugins/AUTO.md # docs/IE_DG/supported_plugins/HETERO.md # docs/IE_DG/supported_plugins/MULTI.md # docs/MO_DG/prepare_model/convert_model/Convert_Model_From_Kaldi.md # docs/MO_DG/prepare_model/convert_model/tf_specific/Convert_YOLO_From_Tensorflow.md # docs/install_guides/installing-openvino-macos.md # docs/install_guides/installing-openvino-windows.md # docs/ops/opset.md # inference-engine/samples/benchmark_app/README.md # inference-engine/tools/benchmark_tool/README.md # thirdparty/ade * Cherry-picked: doc script changes (#8568) * fix openvino-sphinx-theme * add linkcheck target * fix * change version * add doxygen-xfail.txt * fix * AA * fix * fix * fix * fix * fix # Conflicts: # thirdparty/ade * Cherry-pick: Feature/azaytsev/doc updates gna 2021 4 2 (#8567) * Various doc changes * Reformatted C++/Pythob sections. Updated with info from PR8490 * additional fix * Gemini Lake replaced with Elkhart Lake * Fixed links in IGs, Added 12th Gen # Conflicts: # docs/IE_DG/supported_plugins/GNA.md # thirdparty/ade * Cherry-pick: Feature/azaytsev/doc fixes (#8897) * Various doc changes * Removed the empty Learning path topic * Restored the Gemini Lake CPIU list # Conflicts: # docs/IE_DG/supported_plugins/GNA.md # thirdparty/ade * Cherry-pick: sphinx copybutton doxyrest code blocks (#8992) # Conflicts: # thirdparty/ade * Cherry-pick: iframe video enable fullscreen (#9041) # Conflicts: # thirdparty/ade * Cherry-pick: fix untitled titles (#9213) # Conflicts: # thirdparty/ade * Cherry-pick: perf bench graph animation (#9045) * animation * fix # Conflicts: # thirdparty/ade * Cherry-pick: doc pytest (#8888) * docs pytest * fixes # Conflicts: # docs/doxygen/doxygen-ignore.txt # docs/scripts/ie_docs.xml # thirdparty/ade * Cherry-pick: restore deleted files (#9215) * Added new operations to the doc structure (from removed ie_docs.xml) * Additional fixes * Update docs/IE_DG/InferenceEngine_QueryAPI.md Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com> * Update docs/IE_DG/Int8Inference.md Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com> * Update Custom_Layers_Guide.md * Changes according to review comments * doc scripts fixes * Update docs/IE_DG/Int8Inference.md Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com> * Update Int8Inference.md * update xfail * clang format * updated xfail Co-authored-by: Trawinski, Dariusz <dariusz.trawinski@intel.com> Co-authored-by: Nikolay Tyukaev <nikolay.tyukaev@intel.com> Co-authored-by: kblaszczak-intel <karol.blaszczak@intel.com> Co-authored-by: Yury Gorbachev <yury.gorbachev@intel.com> Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com>
2021-12-21 20:26:37 +03:00
1 parent 0c7089acc6
commit 4ae6258bed
@@ -0,0 +1,52 @@
+# Jupyter notebooks autodoc
+
+Auto fetching documentations designed for openvino notebooks tutorials.
+This module is responsible for fetching artifats, in this particular example jupyter tutorial notebooks and converting them to notebook documentation.
+
+## Step 0. Prepare venv
+
+To start new venv based on system interpreter
+
+``` bash
+python -m venv ./venv
+```
+
+Linux:
+
+``` bash
+source ./venv/bin/activate
+```
+
+Windows (cmd):
+
+``` bash
+venv/Scripts/activate
+```
+
+## Step 1. Download requirements
+
+``` bash
+python -m pip install -r requirements.txt
+```
+
+## Step 2. Download personal token from github account
+
+[Tutorial how to do it.](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token). By default configuration you should place it on path ../.secret.
+
+## Step 3. Configure consts to meet project directions
+
+[Consts file](consts.py) contains multiple variables that might differ for different environments.
+
+## Step 4. Add classes with methods to makefile or other executed file
+
+[Main file](main.py) contains example usecases of auto generator for notebooks. Informations placed in [main](main.py) should be further used to add it to makefile and possibly fully automate notebook documentation process.
+
+## Step 5. Run python file (optional)
+
+If step 4 was skipped use command
+
+``` bash
+python main.py
+```
+
+or use any other command that is responsible for generating documentation
@@ -0,0 +1,67 @@
+notebooks_docs = "notebooks.rst"
+
+notebooks_path = "notebooks"
+
+repo_directory = "notebooks"
+
+repo_owner = "openvinotoolkit"
+
+repo_name = "openvino_notebooks"
+
+section_names = ["Getting Started", "Convert & Optimize", "Model Demos", "Model Training", "Live Demos"]
+
+# Templates
+
+binder_template = """
+This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. 
+See the |installation_link| for instructions to run this tutorial locally on Windows, Linux or macOS. 
+To run without installing anything, click the launch binder button.
+
+|binder_link| |github_link|
+
+.. |installation_link| raw:: html
+
+   <a href="https://github.com/{{ owner }}/{{ repo }}#-installation-guide" target="_blank">installation guide</a>
+
+.. |binder_link| raw:: html 
+
+   <a href="https://mybinder.org/v2/gh/{{ owner }}/{{ repo }}/HEAD?filepath={{ folder }}%2F{{ notebook }}%2F{{ notebook }}.ipynb" target="_blank"><img src="https://mybinder.org/badge_logo.svg" alt="Binder"></a>
+
+.. |github_link| raw:: html
+
+   <a href="https://github.com/{{ owner }}/{{ repo }}" target="_blank"><img src="https://badgen.net/badge/icon/github?icon=github&label" alt="Github"></a>
+
+\n
+"""
+no_binder_template = """
+This tutorial is also available as a Jupyter notebook that can be cloned directly from GitHub. 
+See the |installation_link| for instructions to run this tutorial locally on Windows, Linux or macOS.
+
+|github_link|
+
+.. |installation_link| raw:: html
+
+   <a href="https://github.com/{{ owner }}/{{ repo }}#-installation-guide" target="_blank">installation guide</a>
+
+.. |github_link| raw:: html
+
+   <a href="https://github.com/{{ owner }}/{{ repo }}" target="_blank"><img src="https://badgen.net/badge/icon/github?icon=github&label" alt="Github"></a>
+
+\n
+"""
+
+rst_template = """
+OpenVINO notebooks documentation
+================================
+
+{% for section in sections %}
+{{section.name}}
+--------------------------------
+
+.. toctree::
+   :maxdepth: 1
+
+{% for notebook in section.notebooks %}   {{notebook.path}}\n{% endfor %}
+{% endfor %}
+
+"""
@@ -0,0 +1,214 @@
+import argparse
+from pathlib import Path
+from utils import (
+    create_content,
+    add_content_below,
+    load_secret,
+    process_notebook_name,
+    find_latest_artifact,
+    verify_notebook_name,
+    generate_artifact_link,
+    remove_existing,
+    split_notebooks_into_sections,
+)
+from consts import (
+    binder_template,
+    no_binder_template,
+    rst_template,
+    notebooks_path,
+    repo_owner,
+    repo_name,
+    repo_directory,
+    notebooks_docs,
+    section_names
+)
+from notebook import Notebook
+from section import Section
+from io import BytesIO
+from glob import glob
+from jinja2 import Template
+from requests import get
+from zipfile import ZipFile
+import os
+
+
+class NbDownloader:
+    """Class responsible for downloading and extracting notebooks"""
+
+    def __init__(self, secret_path: str) -> None:
+        self.secret = load_secret(secret_path)
+        self.headers = {
+            "Accept": "application/vnd.github.v3+json",
+            "Authorization": f"token {self.secret}",
+        }
+        self.artifact_link = generate_artifact_link(repo_owner, repo_name)
+
+    def default_pipeline(self, path: str = notebooks_path) -> bool:
+        """Default pipeline for fetching, downloading and extracting rst files
+
+        :param path: Path to folder that will contain notebooks. Defaults to notebooks_path.
+        :type path: str
+        :returns: Returns if status is sucessful
+        :rtype: bool
+
+        """
+        artifacts = self.fetch_artifacts()
+        latest_artifact = find_latest_artifact(artifacts)
+        download_link = self.generate_artifact_download_link(latest_artifact)
+        zipfile = self.download_rst_files(download_link)
+        if zipfile.testzip() is None:
+            remove_existing(path)
+        return self.extract_artifacts(zipfile, path=path)
+
+    def fetch_artifacts(self) -> dict:
+        """Fetching artifcats from github actions
+
+        :returns: Artifacts in repo
+        :rtype: dict
+
+        """
+        return get(self.artifact_link, headers=self.headers).json()
+
+    def generate_artifact_download_link(self, artifact_id: int) -> str:
+        """Generate link based on link and latest artifact id containing rst files
+
+        :param artifact_id: Latest artifact id containing rst files
+        :type artifact_id: int
+        :returns: Link to download rst files
+        :rtype: str
+
+        """
+        return f"{self.artifact_link}/{artifact_id}/zip"
+
+    def download_rst_files(self, artifact_download_link: str) -> ZipFile:
+        """Downloading rst files
+
+        :param artifact_download_link: Generated link for downloading rst
+        :type artifact_download_link: str
+        :returns: Zipped archive of rst files
+        :rtype: ZipFile
+
+        """
+        artifact = get(artifact_download_link, headers=self.headers)
+        return ZipFile(BytesIO(artifact.content))
+
+    def extract_artifacts(self, zipfile: ZipFile, path: str) -> bool:
+        """Extracting all artifacts from zipped archive
+
+        :param zipfile: zipped rst files
+        :type zipfile: ZipFile
+        :param path: path to extract files to
+        :type path: str
+        :returns: Returns if status is sucessful
+        :rtype: bool
+
+        """
+        try:
+            zipfile.extractall(path=path)
+            return True
+        except ValueError:
+            return False
+
+
+class NbProcessor:
+    def __init__(self, nb_path: str = notebooks_path):
+        self.nb_path = nb_path
+        notebooks = [
+                Notebook(
+                    name=process_notebook_name(notebook),
+                    path=notebook,
+                )
+                for notebook in os.listdir(self.nb_path)
+                if verify_notebook_name(notebook)
+        ]
+        notebooks = split_notebooks_into_sections(notebooks)
+        self.rst_data = {
+            "sections": [
+                Section(name=section_name, notebooks=section_notebooks)
+                for section_name, section_notebooks in zip(section_names, notebooks)
+            ]
+            
+        }
+        self.binder_data = {
+            "owner": repo_owner,
+            "repo": repo_name,
+            "folder": repo_directory,
+        }
+
+    def fetch_binder_list(self, file_format: str = 'txt') -> list:
+        """Funtion that fetches list of notebooks with binder buttons
+
+        :param file_format: Format of file containing list of notebooks with button. Defaults to 'txt'
+        :type file_format: str
+        :return: List of notebooks conaining binder buttons
+        :rtype: list
+        """
+        list_of_buttons = glob(f"{self.nb_path}/*.{file_format}")
+        if list_of_buttons:
+            with open(list_of_buttons[0]) as file:
+                list_of_buttons = file.read().splitlines()
+            return list_of_buttons
+        else:
+            return []
+
+    def add_binder(self, buttons_list: list,  template_with_binder: str = binder_template, template_without_binder: str = no_binder_template):
+        """Function working as an example how to add binder button to existing rst files
+
+        :param buttons_list: List of notebooks that work on Binder.
+        :type buttons_list: list
+        :param template_with_binder: Template of button added to rst file if Binder is available. Defaults to binder_template.
+        :type template_with_binder: str
+        :param template_without_binder: Template of button added to rst file if Binder isn't available. Defaults to no_binder_template.
+        :type template_without_binder: str
+        :raises FileNotFoundError: In case of failure of adding content, error will appear
+
+        """
+        for notebook in [
+            nb for nb in os.listdir(self.nb_path) if verify_notebook_name(nb)
+        ]:
+            if '-'.join(notebook.split('-')[:-2]) in buttons_list:
+                button_text = create_content(template_with_binder, self.binder_data, notebook)
+                if not add_content_below(button_text, f"{self.nb_path}/{notebook}"):
+                    raise FileNotFoundError("Unable to modify file")
+            else:
+                button_text = create_content(template_without_binder, self.binder_data, notebook)
+                if not add_content_below(button_text, f"{self.nb_path}/{notebook}"):
+                    raise FileNotFoundError("Unable to modify file")
+
+    def render_rst(self, path: str = notebooks_docs, template: str = rst_template):
+        """Rendering rst file for all notebooks
+
+        :param path: Path to notebook main rst file. Defaults to notebooks_docs.
+        :type path: str
+        :param template: Template for default rst page. Defaults to rst_template.
+        :type template: str
+
+        """
+        with open(path, "w+") as nb_file:
+            nb_file.writelines(Template(template).render(self.rst_data))
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('secret', type=Path)
+    parser.add_argument('outdir', type=Path)
+    args = parser.parse_args()
+    secret = args.secret
+    outdir = args.outdir
+    outdir.mkdir(parents=True, exist_ok=True)
+    # Step 1. Create secret file
+    # link: https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token
+    # For this notebooks purpose only repo -> public_repo box is required
+    nbd = NbDownloader(secret)
+    # Step 2. Run default pipeline for downloading
+    if not nbd.default_pipeline(outdir):
+        raise FileExistsError("Files not downloaded")
+    # Step 3. Run processing on downloaded file
+    nbp = NbProcessor(outdir)
+    buttons_list = nbp.fetch_binder_list('txt')
+    nbp.add_binder(buttons_list)
+    nbp.render_rst(outdir.joinpath(notebooks_docs))
+
+
+if __name__ == '__main__':
+    main()
@@ -0,0 +1,4 @@
+class Notebook:
+    def __init__(self, name, path):
+        self.name = name
+        self.path = path
@@ -0,0 +1,2 @@
+jinja2
+requests
@@ -0,0 +1,4 @@
+class Section:
+    def __init__(self, name: str, notebooks: list):
+        self.name = name
+        self.notebooks = notebooks
@@ -0,0 +1,142 @@
+from jinja2 import Template
+from os import path, remove
+from shutil import rmtree
+
+
+def create_content(template: str, notebooks_data: dict, file_name: str):
+    """Filling rst template with data
+
+    :param template: jinja template that will be filled with notebook data
+    :type template: str
+    :param notebooks_data: data structure containing information required to fill template
+    :type notebooks_data: dict
+    :param file_name: file name
+    :type file_name: str
+    :returns: Filled template
+    :rtype: str
+
+    """
+    template = Template(template)
+    notebooks_data["notebook"] = "-".join(file_name.split("-")[:-2])
+    return template.render(notebooks_data)
+
+
+def add_content_below(text: str, path: str, line=3) -> bool:
+    """Add additional content (like binder button) to existing rst file
+
+    :param text: Text that will be added inside rst file
+    :type text: str
+    :param path: Path to modified file
+    :type path: str
+    :param line: Line number that content will be added. Defaults to 3.
+    :type line: int
+    :returns: Informs about success or failure in modifying file
+    :rtype: bool
+
+    """
+    try:
+        with open(path, "r+", encoding="utf-8") as file:
+            current_file = file.readlines()
+            current_file[line:line] = text
+            file.seek(0)
+            file.writelines(current_file)
+            return True
+    except FileNotFoundError:
+        return False
+
+
+def load_secret(path: str = "../.secret") -> str:
+    """Loading secret file
+
+    :param path: Path to secret file. Defaults to "../.secret".
+    :type path: str
+    :returns: Secret key
+    :rtype: str
+
+    """
+    with open(path, "r+") as file:
+        return file.readline().strip()
+
+
+def process_notebook_name(notebook_name: str) -> str:
+    """Processes notebook name
+
+    :param notebook_name: Notebook name by default keeps convention:
+        [3 digit]-name-with-dashes-with-output.rst,
+        example: 001-hello-world-with-output.rst
+    :type notebook_name: str
+    :returns: Processed notebook name,
+        001-hello-world-with-output.rst -> 001. hello world
+    :rtype: str
+
+    """
+    return (
+        notebook_name[:3]
+        + "."
+        + " ".join(notebook_name[4:].split(".")[0].split("-")[:-2])
+    )
+
+
+def find_latest_artifact(artifacts_dict: dict, name: str = "rst_files") -> int:
+    """Finds id of latest artifact that can be downloaded
+
+    :param artifacts_dict: Fetched github actions
+    :type artifacts_dict: dict
+    :param name: Name of searched artifact. Defaults to "rst_files".
+    :type name: str
+    :returns: Id of latest artifact containing rst files
+    :rtype: int
+
+    """
+    return max([r["id"] for r in artifacts_dict["artifacts"] if r["name"] == name])
+
+
+def verify_notebook_name(notebook_name: str) -> bool:
+    """Verification based on notebook name
+
+    :param notebook_name: Notebook name by default keeps convention:
+        [3 digit]-name-with-dashes-with-output.rst,
+        example: 001-hello-world-with-output.rst
+    :type notebook_name: str
+    :returns: Return if notebook meets requirements
+    :rtype: bool
+
+    """
+    return notebook_name[:3].isdigit() and notebook_name[-4:] == ".rst"
+
+
+def generate_artifact_link(owner: str, name: str) -> str:
+    """Generate link for downloading artifacts
+
+    :param owner: Github repo owner name
+    :type owner: str
+    :param name: Github repo name
+    :type name: str
+    :returns: Link to api to download artifacts
+    :rtype: str
+
+    """
+    return f"https://api.github.com/repos/{owner}/{name}/actions/artifacts"
+
+
+def remove_existing(notebooks_path: str) -> None:
+    """Removes file if already existed
+
+    :param notebooks_path: path to file to be removed
+    :type notebooks_path: str
+
+    """
+    if path.exists(notebooks_path):
+        if path.isdir(notebooks_path):
+            rmtree(notebooks_path)
+        else:
+            remove(notebooks_path)
+
+def split_notebooks_into_sections(notebooks: list) -> list:
+    series = [list() for _ in range(5)]
+    for notebook in notebooks:
+        try:
+            series[int(notebook.name[0])].append(notebook)
+        except IndexError:
+            pass
+    return series