From 5fa95ff19d815af8cf6f2dbea7ace34865485cf6 Mon Sep 17 00:00:00 2001 From: Sebastian Golebiewski Date: Thu, 23 Mar 2023 10:12:13 +0100 Subject: [PATCH] DOCS shift to rst - Protecting Deep Learning Model (#16474) --- .../deployment_guide_introduction.md | 8 +- .../deployment/deployment_intro.md | 76 +++++++++------ .../images}/deployment_simplified.svg | 0 .../range_supervision}/img_combined_2.png | 0 .../images/range_supervision}/scheme3.svg | 0 .../quantization/range_supervision/README.md | 95 +++++++++++-------- 6 files changed, 106 insertions(+), 73 deletions(-) rename docs/{img => _static/images}/deployment_simplified.svg (100%) rename {tools/pot/docs/range_supervision/images => docs/_static/images/range_supervision}/img_combined_2.png (100%) rename {tools/pot/docs/range_supervision/images => docs/_static/images/range_supervision}/scheme3.svg (100%) diff --git a/docs/Documentation/deployment_guide_introduction.md b/docs/Documentation/deployment_guide_introduction.md index 4e3162d6de5..6496a3cf494 100644 --- a/docs/Documentation/deployment_guide_introduction.md +++ b/docs/Documentation/deployment_guide_introduction.md @@ -9,12 +9,9 @@ Run and Deploy Locally Deploy via Model Serving -@endsphinxdirective - Once you have a model that meets both OpenVINO™ and your requirements, you can choose how to deploy it with your application. -@sphinxdirective .. panels:: :doc:`Deploy via OpenVINO Runtime ` @@ -30,8 +27,7 @@ Once you have a model that meets both OpenVINO™ and your requirements, you can Deployment via OpenVINO Model Server allows the application to connect to the inference server set up remotely. This way inference can use external resources instead of those available to the application itself. -@endsphinxdirective +Apart from the default deployment options, you may also :doc:`deploy your application for the TensorFlow framework with OpenVINO Integration ` - -Apart from the default deployment options, you may also [deploy your application for the TensorFlow framework with OpenVINO Integration](./openvino_ecosystem_ovtf.md). +@endsphinxdirective \ No newline at end of file diff --git a/docs/OV_Runtime_UG/deployment/deployment_intro.md b/docs/OV_Runtime_UG/deployment/deployment_intro.md index df629a51e97..fc9f4581c37 100644 --- a/docs/OV_Runtime_UG/deployment/deployment_intro.md +++ b/docs/OV_Runtime_UG/deployment/deployment_intro.md @@ -11,47 +11,69 @@ Deploy Application with Deployment Manager Local Distribution Libraries -@endsphinxdirective -> **NOTE**: Note that [running inference in OpenVINO Runtime](../openvino_intro.md) is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration and [develop your application properly](../integrate_with_your_application.md) +.. note:: + Note that :doc:`running inference in OpenVINO Runtime ` is the most basic form of deployment. Before moving forward, make sure you know how to create a proper Inference configuration and :doc:`develop your application properly `. -## Local Deployment Options +Local Deployment Options +######################## - Set a dependency on the existing prebuilt packages, also called "centralized distribution": - - using Debian / RPM packages - a recommended way for Linux operating systems; - - using PIP package manager on PyPI - the default approach for Python-based applications; - - using Docker images - if the application should be deployed as a Docker image, use a pre-built OpenVINO™ Runtime Docker image as a base image in the Dockerfile for the application container image. For more information about OpenVINO Docker images, refer to [Installing OpenVINO on Linux from Docker](../../install_guides/installing-openvino-docker-linux.md) and [Installing OpenVINO on Windows from Docker](../../install_guides/installing-openvino-docker-windows.md). -Furthermore, to customize your OpenVINO Docker image, use the [Docker CI Framework](https://github.com/openvinotoolkit/docker_ci) to generate a Dockerfile and built the image. + + - using Debian / RPM packages - a recommended way for Linux operating systems; + - using PIP package manager on PyPI - the default approach for Python-based applications; + - using Docker images - if the application should be deployed as a Docker image, use a pre-built OpenVINO™ Runtime Docker image as a base image in the Dockerfile for the application container image. For more information about OpenVINO Docker images, refer to :doc:`Installing OpenVINO on Linux from Docker ` and :doc:`Installing OpenVINO on Windows from Docker `. + +Furthermore, to customize your OpenVINO Docker image, use the `Docker CI Framework ` to generate a Dockerfile and built the image. + - Grab a necessary functionality of OpenVINO together with your application, also called "local distribution": - - using [OpenVINO Deployment Manager](deployment-manager-tool.md) - providing a convenient way for creating a distribution package; - - using the advanced [local distribution](local-distribution.md) approach; - - using [a static version of OpenVINO Runtime linked to the final app](https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/static_libaries.md). + + - using :doc:`OpenVINO Deployment Manager ` - providing a convenient way for creating a distribution package; + - using the advanced :doc:`local distribution ` approach; + - using `a static version of OpenVINO Runtime linked to the final app `__. The table below shows which distribution type can be used for what target operating system: -| Distribution type | Operating systems | -|------- ---------- | ----------------- | -| Debian packages | Ubuntu 18.04 long-term support (LTS), 64-bit; Ubuntu 20.04 long-term support (LTS), 64-bit | -| RMP packages | Red Hat Enterprise Linux 8, 64-bit | -| Docker images | Ubuntu 18.04 long-term support (LTS), 64-bit; Ubuntu 20.04 long-term support (LTS), 64-bit; Red Hat Enterprise Linux 8, 64-bit; Windows Server Core base LTSC 2019, 64-bit; Windows 10, version 20H2, 64-bit | -| PyPI (PIP package manager) | See [https://pypi.org/project/openvino/](https://pypi.org/project/openvino/) | -| [OpenVINO Deployment Manager](deployment-manager-tool.md) | All operating systems | -| [Local distribution](local-distribution.md) | All operating systems | -| [Build OpenVINO statically and link to the final app](https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/static_libaries.md) | All operating systems | +.. list-table:: + :header-rows: 1 -## Granularity of Major Distribution Types + * - Distribution type + - Operating systems + * - Debian packages + - Ubuntu 18.04 long-term support (LTS), 64-bit; Ubuntu 20.04 long-term support (LTS), 64-bit + * - RMP packages + - Red Hat Enterprise Linux 8, 64-bit + * - Docker images + - Ubuntu 18.04 long-term support (LTS), 64-bit; Ubuntu 20.04 long-term support (LTS), 64-bit; Red Hat Enterprise Linux 8, 64-bit; Windows Server Core base LTSC 2019, 64-bit; Windows 10, version 20H2, 64-bit + * - PyPI (PIP package manager) + - See https://pypi.org/project/openvino + * - :doc:`OpenVINO Deployment Manager ` + - All operating systems + * - :doc:`Libraries for Local Distribution ` + - All operating systems + * - `Build OpenVINO statically and link to the final app `__ + - All operating systems -The granularity of OpenVINO packages may vary for different distribution types. For example, the PyPI distribution of OpenVINO has a [single 'openvino' package](https://pypi.org/project/openvino/) that contains all the runtime libraries and plugins, while a [local distribution](local-distribution.md) is a more configurable type providing higher granularity. Below are important details of the set of libraries included in the OpenVINO Runtime package: -![](../../img/deployment_simplified.svg) +Granularity of Major Distribution Types +####################################### -- The main library `openvino` is used by users' C++ applications to link against with. The library provides all OpenVINO Runtime public APIs, including both API 2.0 and the previous Inference Engine and nGraph APIs. For C language applications, `openvino_c` is additionally required for distribution. -- The "optional" plugin libraries like `openvino_intel_cpu_plugin` (matching the `openvino_.+_plugin` pattern) are used to provide inference capabilities on specific devices or additional capabilities like [Hetero Execution](../hetero_execution.md) and [Multi-Device Execution](../multi_device.md). -- The "optional" plugin libraries like `openvino_ir_frontend` (matching `openvino_.+_frontend`) are used to provide capabilities to read models of different file formats such as OpenVINO IR, TensorFlow, ONNX, and PaddlePaddle. +The granularity of OpenVINO packages may vary for different distribution types. For example, the PyPI distribution of OpenVINO has a `single 'openvino' package `__ that contains all the runtime libraries and plugins, while a :doc:`local distribution ` is a more configurable type providing higher granularity. Below are important details of the set of libraries included in the OpenVINO Runtime package: + +.. image:: _static/images/deployment_simplified.svg + + +- The main library ``openvino`` is used by users' C++ applications to link against with. The library provides all OpenVINO Runtime public APIs, including both API 2.0 and the previous Inference Engine and nGraph APIs. For C language applications, ``openvino_c`` is additionally required for distribution. +- The "optional" plugin libraries like ``openvino_intel_cpu_plugin`` (matching the ``openvino_.+_plugin`` pattern) are used to provide inference capabilities on specific devices or additional capabilities like :doc:`Hetero Execution ` and :doc:`Multi-Device Execution `. +- The "optional" plugin libraries like ``openvino_ir_frontend`` (matching ``openvino_.+_frontend``) are used to provide capabilities to read models of different file formats such as OpenVINO IR, TensorFlow, ONNX, and PaddlePaddle. Here the term "optional" means that if the application does not use the capability enabled by the plugin, the plugin library or a package with the plugin is not needed in the final distribution. -Building a local distribution will require more detailed information, and you will find it in the dedicated [Libraries for Local Distribution](local-distribution.md) article. +Building a local distribution will require more detailed information, and you will find it in the dedicated :doc:`Libraries for Local Distribution ` article. -> **NOTE**: Depending on your target OpenVINO devices, the following configurations might be needed for deployed machines: [Configurations for GPU](../../install_guides/configurations-for-intel-gpu.md), [Configurations for GNA](../../install_guides/configurations-for-intel-gna.md). +.. note:: + + Depending on your target OpenVINO devices, the following configurations might be needed for deployed machines: :doc:`Configurations for GPU `, :doc:`Configurations for GNA `. + +@endsphinxdirective \ No newline at end of file diff --git a/docs/img/deployment_simplified.svg b/docs/_static/images/deployment_simplified.svg similarity index 100% rename from docs/img/deployment_simplified.svg rename to docs/_static/images/deployment_simplified.svg diff --git a/tools/pot/docs/range_supervision/images/img_combined_2.png b/docs/_static/images/range_supervision/img_combined_2.png similarity index 100% rename from tools/pot/docs/range_supervision/images/img_combined_2.png rename to docs/_static/images/range_supervision/img_combined_2.png diff --git a/tools/pot/docs/range_supervision/images/scheme3.svg b/docs/_static/images/range_supervision/scheme3.svg similarity index 100% rename from tools/pot/docs/range_supervision/images/scheme3.svg rename to docs/_static/images/range_supervision/scheme3.svg diff --git a/tools/pot/openvino/tools/pot/algorithms/quantization/range_supervision/README.md b/tools/pot/openvino/tools/pot/algorithms/quantization/range_supervision/README.md index 0385c31e4ab..de7b1695054 100644 --- a/tools/pot/openvino/tools/pot/algorithms/quantization/range_supervision/README.md +++ b/tools/pot/openvino/tools/pot/algorithms/quantization/range_supervision/README.md @@ -1,70 +1,85 @@ # Experimental: Protecting Deep Learning Model through Range Supervision ("RangeSupervision") {#pot_ranger_README} -## Introduction +@sphinxdirective + +Introduction +#################### Deep neural network find applications in many scenarios where the prediction is a critical component for safety-relevant decisions. Such workloads can benefit from additional protection against underlying errors. For example, memory bit flips (**"soft errors"** originating, e.g., from external radiation or internal electrical disturbances within the circuitry) in der platform hosting the network inference can corrupt the learned network parameters and lead to incorrect predictions. Typically, errors resulting in very large parameter values have a more drastic impact on the network behavior. **The range supervision algorithm ("RangeSupervision") described here establishes and inserts additional protection layers after already present activation layers**. Those layers truncate values that are found to be out of an expected activation range in order to mitigate the traces of potential platform errors. They do so during inference by applying a *clamp* operation to any activation *x* in the input to the RangeSupervision layer, - \f[ - x = clamp(x ; T_{low}, T_{up}) = min(max(x, T_{low}), T_{high}) - \f] - where \f$T_{low}\f$ and \f$T_{up}\f$ are the lower and upper bounds for the particular protection layer, respectively. -The process flow follows the diagram [Fig 1](#schematic-supervision). Starting from the internal representation (IR) of an OpenVINO model, the POT RangeSupervision algorithm is called to **add protection layers into the model graph**. This step requires **appropriate threshold values that are automatically extracted from a specified test dataset**. The result is an IR representation of the model with additional "RangeSupervision" layers after each supported activation layer. The original and the modified model can be called in the same way through the OpenVINO inference engine to evaluate the impact on accuracy, performance, and dependability in the presence of potential soft errors (for example using the *benchmark_app* and *accuracy_checker* functions). **The algorithm is designed to provide efficient protection at negligible performance overhead or accuracy impact in the absence of faults.** Bound extraction is a one-time effort and the protected IR model returned by the RangeSupervision algorithm can be used independently from there on. No changes in the learned parameters of the network are needed. +.. math:: - + x = clamp(x ; T_{low}, T_{up}) = min(max(x, T_{low}), T_{high}) + + +where :math:`T_{low}` and :math:`T_{up}` are the lower and upper bounds for the particular protection layer, respectively. +The process flow follows the diagram :ref:`Fig 1 `. Starting from the internal representation (IR) of an OpenVINO model, the POT RangeSupervision algorithm is called to **add protection layers into the model graph**. This step requires **appropriate threshold values that are automatically extracted from a specified test dataset**. The result is an IR representation of the model with additional "RangeSupervision" layers after each supported activation layer. The original and the modified model can be called in the same way through the OpenVINO inference engine to evaluate the impact on accuracy, performance, and dependability in the presence of potential soft errors (for example using the *benchmark_app* and *accuracy_checker* functions). **The algorithm is designed to provide efficient protection at negligible performance overhead or accuracy impact in the absence of faults.** Bound extraction is a one-time effort and the protected IR model returned by the RangeSupervision algorithm can be used independently from there on. No changes in the learned parameters of the network are needed. + +.. _schematic-supervision: + +.. image:: _static/images/range_supervision/scheme3.svg + :alt: Schematic -@anchor schematic -![Schematic](../../../../../../docs/range_supervision/images/scheme3.svg) *Fig 1: Schematic of RangeSupervision process flow.* - -### Supported activation layers +Supported activation layers ++++++++++++++++++++++++++++ The following activation layers are currently supported for range supervision: - - `ReLU` - - `Swish` - - `PReLU` - - `Elu` - - `Gelu` - - `Sigmoid` - - `Tanh` - +- `ReLU` +- `Swish` +- `PReLU` +- `Elu` +- `Gelu` +- `Sigmoid` +- `Tanh` + This means that any activation layer of one of the above types, that the model under consideration contains, will be protected with an appropriate subsequent RangeSupervision layer. -## Usage -RangeSupervision protection can be used the same way as [DefaultQuantization](@ref pot_default_quantization_usage) method. +Usage +#################### + +RangeSupervision protection can be used the same way as :doc:`DefaultQuantization ` method. + +Algorithm configuration ++++++++++++++++++++++++ -### Algorithm configuration Algorithm has a minimal configuration. Below is an example of such configuration: -```json -{ - "name": "RangeSupervision", - "params": { - "stat_subset_size": 300 - "stat_batch_size": 1 - } - -} -``` +.. code-block:: json -The protected model will be saved in IR format in a new folder ``./results/\_RangeSupervision/...`` . + { + "name": "RangeSupervision", + "params": { + "stat_subset_size": 300 + "stat_batch_size": 1 + } + } + + +The protected model will be saved in IR format in a new folder ``./results/\_RangeSupervision/...``. Mandatory parameters: -- `"stat_subset_size"`: This parameter defines *how many images* of the specified dataset in "engine: config" are used to extract the bounds (images are randomly chosen if a subset is chosen). This value is set to **300** by default. The more images are selected for the bound generation, the more accurate the estimation of an out-of-bound event will be, at the cost of increasing extraction time. -## Example of RangeSupervision results +- ``"stat_subset_size"``: This parameter defines *how many images* of the specified dataset in "engine: config" are used to extract the bounds (images are randomly chosen if a subset is chosen). This value is set to **300** by default. The more images are selected for the bound generation, the more accurate the estimation of an out-of-bound event will be, at the cost of increasing extraction time. + +Example of RangeSupervision results +################################### The following example shows a traffic camera image and predicted objects using a Yolov3 pre-trained on the Coco dataset. A single weight fault was injected in a randomly chosen convolution layer of YOLO, flipping the most significant bit of the selected network parameter. If range supervision is applied, the original network performance is recovered despite the presence of the fault. +.. image:: _static/images/range_supervision/img_combined_2.png -![](../../../../../../docs/range_supervision/images/img_combined_2.png) - *Fig 2: Example of fault mitigation via range supervision.* -## Resources: +Additional Resources +#################### + +- Z. Chen, G. Li, and K. Pittabiraman, "A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction", 2020. https://arxiv.org/abs/2003.13874 +- F. Geissler, Q. Syed, S. Roychowdhury, A. Asgari, Y. Peng, A. Dhamasia, R. Graefe, K. Pattabiraman, and M. Paulitsch, "Towards a Safety Case for Hardware Fault Tolerance in Convolutional Neural Networks Using Activation Range Supervision", 2021. https://arxiv.org/abs/2108.07019 + + @endsphinxdirective - - Z. Chen, G. Li, and K. Pittabiraman, "A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction", 2020. https://arxiv.org/abs/2003.13874 - - F. Geissler, Q. Syed, S. Roychowdhury, A. Asgari, Y. Peng, A. Dhamasia, R. Graefe, K. Pattabiraman, and M. Paulitsch, "Towards a Safety Case for Hardware Fault Tolerance in Convolutional Neural Networks Using Activation Range Supervision", 2021. https://arxiv.org/abs/2108.07019