[POT] Remove tpe algo (#8929)
* remove tpe * delete unused object_dump.py * delete tpe reference * tpe removal in optimization * deleted benchmark * docs changes * tunable quantization changes * mistype in sentence
This commit is contained in:
Milana Shhanukova
@@ -18,7 +18,6 @@ features:
|
||||
* [API](openvino/tools/pot/api/README.md) that helps to apply optimization methods within a custom inference script written with OpenVINO Python* API.
|
||||
* Symmetric and asymmetric quantization schemes. For details, see the [Quantization](openvino/tools/pot/algorithms/quantization/README.md) section.
|
||||
* Per-channel quantization for Convolutional and Fully-Connected layers.
|
||||
* Global optimization of post-training quantization parameters using the [Tree-Structured Parzen Estimator](openvino/tools/pot/optimization/tpe/README.md).
|
||||
|
||||
The tool is aimed to fully automate the model transformation process without a need to change the model on the user's side.
|
||||
The POT is available only in the Intel® distribution of OpenVINO™ toolkit and is not opensourced. For details
|
||||
|
||||
@@ -8,7 +8,7 @@ Post-Training Optimization Tool includes standalone command-line tool and Python
|
||||
## Key features:
|
||||
|
||||
* Two supported post-training quantization algorithms: fast [DefaultQuantization](openvino/tools/pot/algorithms/quantization/default/README.md) and precise [AccuracyAwareQuantization](openvino/tools/pot/algorithms/quantization/accuracy_aware/README.md), as well as multiple experimental methods.
|
||||
* Global optimization of post-training quantization parameters using [Tree-structured Parzen Estimator](openvino/tools/pot/optimization/tpe/README.md).
|
||||
|
||||
* Symmetric and asymmetric quantization schemes. For more details, see the [Quantization](openvino/tools/pot/algorithms/quantization/README.md) section.
|
||||
* Per-channel quantization for Convolutional and Fully-Connected layers.
|
||||
* Multiple domains: Computer Vision, Recommendation Systems.
|
||||
|
||||
@@ -124,151 +124,5 @@ accuracy-aware scenario.
|
||||
|
||||
If you do not achieve the desired accuracy and performance after applying the
|
||||
`AccuracyAwareQuantization` algorithm or you need an accurate fully-quantized model,
|
||||
we recommend either using layer-wise hyperparameters tuning with TPE or using
|
||||
we recommend using layer-wise hyperparameters tuning using
|
||||
Quantization-Aware training from [the supported frameworks](LowPrecisionOptimizationGuide.md).
|
||||
|
||||
## Layer-Wise Hyperparameters Tuning Using TPE
|
||||
|
||||
As the last step in post-training optimization, you may try layer-wise hyperparameter
|
||||
tuning using TPE, which stands for Tree of Parzen Estimators hyperparameter optimizer
|
||||
that searches through available configurations trying to find an optimal one.
|
||||
For post-training optimization, TPE assigns multiple available configuration
|
||||
options to choose from for every layer and by evaluating different sets of parameters,
|
||||
it creates a probabilistic model of their impact on accuracy and latency to
|
||||
iteratively find an optimal one.
|
||||
|
||||
You can run TPE with any combination of parameters in `tuning_scope`, but it is
|
||||
recommended to use one of two configurations described below. It is recommended to first try
|
||||
Range Estimator Configuration. If this configuration will not be able to reach accuracy
|
||||
target then it is recommended to run Layer Configuration. If for some reason,
|
||||
like HW failure or power shutdown, TPE trials stop before completion, you can
|
||||
rerun them starting from the last trial by changing `trials_load_method`
|
||||
from `cold_start` to `warm_start` as long as logs from the previous execution are available.
|
||||
|
||||
> **NOTE**: TPE requires many iterations to converge to an optimal solution, and
|
||||
> it is recommended to run it for at least 200 iterations. Because every iteration
|
||||
> requires evaluation of a generated model , which means accuracy measurements on a
|
||||
> dataset and latency measurements using benchmark, this process may take from
|
||||
> 24 hours up to few days to complete, depending on a model.
|
||||
> To run this configuration on multiple machines and reduce the execution time,
|
||||
> see [Multi-node](../openvino/tools/pot/optimization/tpe/multinode.md).
|
||||
|
||||
### Range Estimator Configuration
|
||||
|
||||
To run TPE with range estimator tuning, use the following configuration:
|
||||
```json
|
||||
"optimizer": {
|
||||
"name": "Tpe",
|
||||
"params": {
|
||||
"max_trials": 200,
|
||||
"trials_load_method": "cold_start",
|
||||
"accuracy_loss": 0.1,
|
||||
"latency_reduce": 1.5,
|
||||
"accuracy_weight": 1.0,
|
||||
"latency_weight": 0.0,
|
||||
"benchmark": {
|
||||
"performance_count": false,
|
||||
"batch_size": 1,
|
||||
"nthreads": 8,
|
||||
"nstreams": 1,
|
||||
"nireq": 1,
|
||||
"api_type": "async",
|
||||
"niter": 1,
|
||||
"duration_seconds": 30,
|
||||
"benchmark_app_dir": "<path to benchmark_app>" // Path to benchmark_app If not specified, Python base benchmark will be used. Use benchmark_app to reduce jitter in results.
|
||||
}
|
||||
}
|
||||
},
|
||||
"compression": {
|
||||
"target_device": "ANY",
|
||||
"algorithms": [
|
||||
{
|
||||
"name": "ActivationChannelAlignment",
|
||||
"params": {
|
||||
"stat_subset_size": 300
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "TunableQuantization",
|
||||
"params": {
|
||||
"stat_subset_size": 300,
|
||||
"preset": "performance",
|
||||
"tuning_scope": ["range_estimator"],
|
||||
"estimator_tuning_scope": ["preset", "outlier_prob"],
|
||||
"outlier_prob_choices": [1e-3, 1e-4, 1e-5]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "FastBiasCorrection",
|
||||
"params": {
|
||||
"stat_subset_size": 300
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This configuration searches for optimal preset for `range_estimator` and optimal
|
||||
outlier probability for quantiles for every layer. Because this configuration
|
||||
only changes final values provided to [FakeQuantize]((https://docs.openvinotoolkit.org/latest/_docs_ops_quantization_FakeQuantize_1.html)) layers, changes in parameters
|
||||
do not impact inference latency, thus we set `latency_weight` to 0 to prevent
|
||||
jitter in benchmark results to negatively impact model evaluation. Experiments
|
||||
show that this configuration can give much better accuracy then the approach of
|
||||
just changing `range_estimator` configuration globally.
|
||||
|
||||
### Layer Configuration
|
||||
|
||||
To run TPE with layer tuning, use the following configuration:
|
||||
```json
|
||||
"optimizer": {
|
||||
"name": "Tpe",
|
||||
"params": {
|
||||
"max_trials": 200,
|
||||
"trials_load_method": "cold_start",
|
||||
"accuracy_loss": 0.1,
|
||||
"latency_reduce": 1.5,
|
||||
"accuracy_weight": 1.0,
|
||||
"latency_weight": 1.0,
|
||||
"benchmark": {
|
||||
"performance_count": false,
|
||||
"batch_size": 1,
|
||||
"nthreads": 8,
|
||||
"nstreams": 1,
|
||||
"nireq": 1,
|
||||
"api_type": "async",
|
||||
"niter": 1,
|
||||
"duration_seconds": 30,
|
||||
"benchmark_app_dir": "<path to benchmark_app>" // Path to benchmark_app If not specified, Python base benchmark will be used. Use benchmark_app to reduce jitter in results.
|
||||
}
|
||||
}
|
||||
},
|
||||
"compression": {
|
||||
"target_device": "ANY",
|
||||
"algorithms": [
|
||||
{
|
||||
"name": "ActivationChannelAlignment",
|
||||
"params": {
|
||||
"stat_subset_size": 300
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "TunableQuantization",
|
||||
"params": {
|
||||
"stat_subset_size": 300,
|
||||
"preset": "performance",
|
||||
"tuning_scope": ["layer"]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "FastBiasCorrection",
|
||||
"params": {
|
||||
"stat_subset_size": 300
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This configuration is similar to `AccuracyAwareQuantization`, because it also
|
||||
tries to revert quantized layers back to floating-point precision, but uses a
|
||||
different algorithm to choose layers, which can lead to better results.
|
||||
|
||||
@@ -79,7 +79,6 @@ It can happen due to the following reasons:
|
||||
|
||||
Quantization time depends on multiple factors such as the size of the model and the dataset. It also depends on the algorithm:
|
||||
the [DefaultQuantization](../openvino/tools/pot/algorithms/quantization/default/README.md) algorithm takes less time than the [AccuracyAwareQuantization](../openvino/tools/pot/algorithms/quantization/accuracy_aware/README.md) algorithm.
|
||||
The [Tree-Structured Parzen Estimator (TPE)](../openvino/tools/pot/optimization/tpe/README.md) algorithm might take even more time.
|
||||
The following configuration parameters also impact the quantization time duration
|
||||
(see details in [Post-Training Optimization Best Practices](BestPractices.md)):
|
||||
- `use_fast_bias`: when set to `false`, it increases the quantization time
|
||||
|
||||
@@ -30,7 +30,6 @@ from .algorithms.quantization.weight_bias_correction.algorithm import (
|
||||
from .algorithms.sparsity.magnitude_sparsity.algorithm import MagnitudeSparsity
|
||||
from .algorithms.sparsity.default.algorithm import WeightSparsity
|
||||
from .algorithms.sparsity.default.base_algorithm import BaseWeightSparsity
|
||||
from .optimization.tpe.base_algorithm import Tpe
|
||||
from .algorithms.quantization.overflow_correction.algorithm import OverflowCorrection
|
||||
from .algorithms.quantization.ranger.algorithm import Ranger
|
||||
|
||||
@@ -47,7 +46,6 @@ QUANTIZATION_ALGORITHMS = [
|
||||
'AccuracyAwareCommon',
|
||||
'INT4MixedQuantization',
|
||||
'TunableQuantization',
|
||||
'Tpe',
|
||||
'QuantNoiseEstimator',
|
||||
'OutlierChannelSplitting',
|
||||
'WeightBiasCorrection',
|
||||
|
||||
@@ -57,11 +57,9 @@ class Algorithm(ABC):
|
||||
:return: None
|
||||
"""
|
||||
|
||||
def get_parameter_meta(self, _model, _optimizer_state):
|
||||
def get_parameter_meta(self, _model):
|
||||
""" Get parameters metadata
|
||||
:param _model: model to get parameters for
|
||||
:param _optimizer_state: dictionary describing optimizer state that allow to tune created search space
|
||||
differently for different optimizer states
|
||||
:return params_meta: metadata of optional parameters
|
||||
"""
|
||||
return []
|
||||
|
||||
@@ -22,10 +22,6 @@ wide range of DNN models:
|
||||
of performance improvement. It may require more time for quantization. For details, see the
|
||||
[AccuracyAwareQuantization Algorithm](@ref pot_compression_algorithms_quantization_accuracy_aware_README) documentation.
|
||||
|
||||
* **Tree-Structured Parzen Estimator (TPE)** similarly to **AccuracyAwareQuantization** enables remaining at a predefined range of accuracy drop at the cost
|
||||
of performance improvement, but additionally tries to provide best possible performance improvement. It requires even more time for quantization than **AccuracyAwareQuantization**,
|
||||
but may lead to better performance improvement. For details, see the [Tree-Structured Parzen Estimator (TPE)](@ref pot_compression_optimization_tpe_README) documentation.
|
||||
|
||||
## Quantization Formula
|
||||
|
||||
Quantization is parametrized by clamping the range and the number of quantization levels:
|
||||
|
||||
+1
-20
@@ -2,16 +2,14 @@
|
||||
|
||||
## Overview
|
||||
|
||||
TunableQuantization algorithm is a modified version (to support hyperparameters setting by [Tree-Structured Parzen Estimator (TPE)](../../../optimization/tpe/README.md)) of the vanilla **MinMaxQuantization** quantization method that automatically inserts [FakeQuantize](https://docs.openvinotoolkit.org/latest/_docs_ops_quantization_FakeQuantize_1.html) operations into the model graph based on the specified target hardware and initializes them using statistics collected on the calibration dataset.
|
||||
TunableQuantization algorithm is a modified version of the vanilla **MinMaxQuantization** quantization method that automatically inserts [FakeQuantize](https://docs.openvinotoolkit.org/latest/_docs_ops_quantization_FakeQuantize_1.html) operations into the model graph based on the specified target hardware and initializes them using statistics collected on the calibration dataset.
|
||||
It is recommended to be run as a part of an optimization pipeline similar to the one used in [**DefaultQuantization**](../default/README.md):
|
||||
* ActivationChannelAlignment - Used as a preliminary step before quantization and allows you to align ranges of output activations of Convolutional layers in order to reduce the quantization error.
|
||||
* TunableQuantization - Used instead of **MinMaxQuantization** to allow tuning using [Tree-Structured Parzen Estimator (TPE)](@ref pot_compression_optimization_tpe_README).
|
||||
* FastBiasCorrection - Adjusts biases of Convolutional and Fully-Connected layers based on the quantization error of the layer in order to make the overall error unbiased.
|
||||
|
||||
## Parameters
|
||||
The recommended parameters sets for TunableQuantization are provided in [post-training optimization best practices](@ref pot_docs_BestPractices).
|
||||
Here we present the complete reference of parameters with their low-level meaning.
|
||||
It is intended for advanced users who would like to experiment with new [Tree-Structured Parzen Estimator (TPE)](../../../optimization/tpe/README.md)'s search spaces.
|
||||
|
||||
The algorithm accepts the following parameters:
|
||||
- `"preset"` - preset which controls the quantization mode (symmetric and asymmetric). It can take two values:
|
||||
@@ -21,23 +19,6 @@ The algorithm accepts the following parameters:
|
||||
for quantization of neural networks which has both negative and positive input values in quantizing operations.
|
||||
- `"stat_subset_size"` - size of subset to calculate activations statistics used for quantization. The whole dataset
|
||||
is used if no parameter specified. We recommend using not less than 300 samples.
|
||||
- `"tuning_scope"` determines which quantization configurations will be returned to [Tree-Structured Parzen Estimator (TPE)](../../../optimization/tpe/README.md) as viable options and can be a list of any combination of the following values:
|
||||
- `"bits"` - layer-wise choice of precision (e.g. INT8, INT4) if hardware supports
|
||||
- `"mode"` - layer-wise choice of symmetric or asymmetric quantization mode
|
||||
- `"range_estimator"` - layer-wise choice of configuration of the algorithm used to estimate min/max FP32 values for the layer
|
||||
- `"layer"` - adds to the possible quantization configurations option that specific layer will not be quantized
|
||||
- `"estimator_tuning_scope"` determines which parameters of the FP32 range estimator will be tuned. This parameter is only necessary when `"range_estimator"` is part of `"tuning_scope"`. Value is a list of any combination of the following values:
|
||||
- `"preset"` - choice between two presets: default (using min/max functions) and quantile (removes outlier values out of FP32 range)
|
||||
- `"type"` - choice similar to preset, but a bit more granular i.e. separate configuration of min and max functions for every layer
|
||||
- `"aggregator"` - choice of function used to aggregate min/max values from all input data samples
|
||||
- `"outlier_prob"` - enables tuning of outlier probability value for quantile configurations – outlier probability specify what fraction of FP32 values in input will be considered as out of range and will get saturated min/max value
|
||||
- `"outlier_prob_choices"` - list of `"outlier_prob"` values to use when tuning `"outlier_prob"` parameter. This parameter is only necessary when `"outlier_prob"` is part of `"estimator_tuning_scope"`.
|
||||
|
||||
List of quantization configurations that will be returned to [Tree-Structured Parzen Estimator (TPE)](../../../optimization/tpe/README.md) as viable options is done through derivation process. This derivation is done by creating a list of all available quantization configurations supported by target hardware and then filtering it using base configuration (either from `"preset"` or previous best result) and `"tuning_scope"`. Filtering is done by choosing from all available options only those that differ from base configuration only on values of variables specified in `"tuning_scope"`.
|
||||
|
||||
The selection of whether to use `"preset"` or previous best result as base configuration depends on [Tree-Structured Parzen Estimator (TPE)](../../../optimization/tpe/README.md)'s `"trials_load_method"`:
|
||||
- `"cold_start - preset"` determines base quantization configuration,
|
||||
- `"fine_tune - preset"` option is ignored and quantization configuration used to achieve the best result in previous run is used as base quantization configuration.
|
||||
|
||||
Below is a fragment of the configuration file that shows overall structure of parameters for this algorithm.
|
||||
|
||||
|
||||
+4
-12
@@ -54,14 +54,9 @@ class TunableQuantization(MinMaxQuantization):
|
||||
|
||||
return activations_stats_layout
|
||||
|
||||
def get_parameter_meta(self, model, optimizer_state):
|
||||
def get_parameter_meta(self, model):
|
||||
param_grid = []
|
||||
if 'range_estimator' in self._config.tuning_scope:
|
||||
for variable in self._config.estimator_tuning_scope:
|
||||
self._config.tuning_scope.append('estimator_' + variable)
|
||||
config = deepcopy(self._config)
|
||||
if optimizer_state['first_iteration'] or optimizer_state['fully_quantized']:
|
||||
config['tuning_scope'] = []
|
||||
|
||||
hardware_config = load_hardware_config(config)
|
||||
model = deepcopy(model)
|
||||
@@ -82,17 +77,14 @@ class TunableQuantization(MinMaxQuantization):
|
||||
if 'activations' in node_config:
|
||||
node_config['activations'] = ut.append_estimator_configs(
|
||||
node_config['activations'], False, config,
|
||||
self.params[node_name] if not optimizer_state['fully_quantized']
|
||||
and node_name in self.params else None)
|
||||
self.params[node_name] if node_name in self.params else None)
|
||||
if 'weights' in node_config:
|
||||
node_config['weights'] = ut.append_estimator_configs(
|
||||
node_config['weights'], True, config,
|
||||
self.params[node_name] if not optimizer_state['fully_quantized']
|
||||
and node_name in self.params else None)
|
||||
self.params[node_name] if node_name in self.params else None)
|
||||
|
||||
for node_name, node_config in nodes_config.items():
|
||||
op_config = ut.get_quantize_op_config(node_config, config,
|
||||
self.params[node_name] if not optimizer_state['fully_quantized']
|
||||
and node_name in self.params else None)
|
||||
self.params[node_name] if node_name in self.params else None)
|
||||
param_grid.append((node_name, 'choice', op_config))
|
||||
return param_grid
|
||||
|
||||
@@ -13,7 +13,6 @@ from openvino.tools.pot.data_loaders.creator import create_data_loader
|
||||
from openvino.tools.pot.engines.creator import create_engine
|
||||
from openvino.tools.pot.graph import load_model, save_model
|
||||
from openvino.tools.pot.graph.model_utils import compress_model_weights
|
||||
from openvino.tools.pot.optimization.optimizer_selector import OPTIMIZATION_ALGORITHMS
|
||||
from openvino.tools.pot.pipeline.initializer import create_pipeline
|
||||
from openvino.tools.pot.utils.logger import init_logger, get_logger
|
||||
from openvino.tools.pot.utils.telemetry import start_session_telemetry, end_session_telemetry
|
||||
@@ -80,16 +79,6 @@ def _update_config_path(args):
|
||||
args.config = os.path.join(config_template_folder, 'accuracy_aware_quantization_template.json')
|
||||
|
||||
|
||||
def print_optimizer_config(config):
|
||||
# log algorithms settings
|
||||
optimizer_string = 'Optimizer: {}'.format(config.name)
|
||||
optimizer_string += '\n Parameters:'
|
||||
for name, value in config.params.items():
|
||||
optimizer_string += '\n\t{: <27s}: {}'.format(name, value)
|
||||
optimizer_string += '\n {}'.format('=' * 75)
|
||||
logger.info(optimizer_string)
|
||||
|
||||
|
||||
def print_algo_configs(config):
|
||||
# log algorithms settings
|
||||
configs_string = 'Creating pipeline:'
|
||||
@@ -121,12 +110,7 @@ def optimize(config):
|
||||
pipeline = create_pipeline(
|
||||
config.compression.algorithms, engine, 'CLI')
|
||||
|
||||
if 'optimizer' in config:
|
||||
print_optimizer_config(config.optimizer)
|
||||
optimizer = OPTIMIZATION_ALGORITHMS.get(config.optimizer.name)(config.optimizer, pipeline, engine)
|
||||
compressed_model = optimizer.run(model)
|
||||
else:
|
||||
compressed_model = pipeline.run(model)
|
||||
compressed_model = pipeline.run(model)
|
||||
|
||||
if not config.model.keep_uncompressed_weights:
|
||||
compress_model_weights(compressed_model)
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
@@ -1,207 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import os
|
||||
from datetime import datetime
|
||||
from tempfile import gettempdir
|
||||
from importlib import import_module
|
||||
from subprocess import check_output
|
||||
import numpy as np
|
||||
from openvino.inference_engine import IENetwork, IECore # pylint: disable=E0611
|
||||
from .infer_request_wrap import InferRequestsQueue
|
||||
from ..graph.model_utils import save_model
|
||||
from ..utils.logger import get_logger
|
||||
from ..utils.utils import create_tmp_dir
|
||||
|
||||
logger = get_logger(__name__)
|
||||
benchmark_cfg = {'nireq': 0, 'nstreams': None, 'nthreads': None, 'performance_count': False,
|
||||
'batch_size': 0, 'niter': 100, 'duration_seconds': 30, 'api_type': 'async',
|
||||
'cpu_bind_thread': 'YES', 'bench_test_number': 0, 'benchmark_cpp': False,
|
||||
'benchmark_app_dir':""}
|
||||
__MODEL_PATH__ = create_tmp_dir(gettempdir())
|
||||
|
||||
def set_benchmark_config(cfg):
|
||||
for key in cfg.keys():
|
||||
if key in benchmark_cfg.keys():
|
||||
benchmark_cfg[key] = cfg[key]
|
||||
else:
|
||||
logger.error('Illegal key {}'.format(key))
|
||||
|
||||
benchmark_cfg['benchmark_cpp'] = False
|
||||
if 'benchmark_app_dir' in cfg and benchmark_cfg['benchmark_app_dir']:
|
||||
if os.path.exists(benchmark_cfg['benchmark_app_dir']):
|
||||
benchmark_cfg['benchmark_cpp'] = True
|
||||
return
|
||||
logger.warning("Fail to find benchmark_app in {}.".format(benchmark_cfg['benchmark_app_dir']))
|
||||
openvino_model = import_module(IENetwork.__module__)
|
||||
path = os.path.dirname(openvino_model.__file__)
|
||||
benchmark_cpp_app = path + "/../../../../../benchmark_app"
|
||||
if os.path.exists(benchmark_cpp_app):
|
||||
benchmark_cfg['benchmark_cpp'] = True
|
||||
benchmark_cfg['benchmark_app_dir'] = benchmark_cpp_app
|
||||
logger.info("Using benchamr_app in {}.".format(benchmark_cpp_app))
|
||||
return
|
||||
logger.info("Using python interface for benchmark.")
|
||||
|
||||
def benchmark_embedded(model=None, mf=None, api_type=None, duration_seconds=0, config=None):
|
||||
""" Perform benchmark with dummy inputs, return inference requests's latency' mesurement result.
|
||||
:param model: model to be benchmarked.
|
||||
:param mf: if model is not provided, model file name should be provided.
|
||||
:param api_type: sync or async.
|
||||
:param duration_seconds: durations in seconds.
|
||||
:param config: config dict to be changed.
|
||||
:return: latency metrics.
|
||||
"""
|
||||
if config:
|
||||
set_benchmark_config(config)
|
||||
if duration_seconds != 0:
|
||||
benchmark_cfg['duration_seconds'] = duration_seconds
|
||||
if api_type:
|
||||
benchmark_cfg['api_type'] = api_type
|
||||
if model:
|
||||
model_name = 'tmp_benmark_model_{}'.format(benchmark_cfg['bench_test_number'])
|
||||
paths = save_model(model, __MODEL_PATH__.name, model_name)
|
||||
path_to_model_file = paths[0]['model']
|
||||
if mf:
|
||||
path_to_model_file = mf.strip()
|
||||
if '.xml' not in path_to_model_file:
|
||||
logger.error('{} is not an xml file.'.format(path_to_model_file))
|
||||
return None
|
||||
|
||||
if not os.path.exists(path_to_model_file):
|
||||
logger.error('{} does not exist.'.format(path_to_model_file))
|
||||
return None
|
||||
if benchmark_cfg['benchmark_cpp']:
|
||||
return benchmark_embedded_cpp_app(path_to_model_file)
|
||||
return benchmark_embedded_python_api(path_to_model_file)
|
||||
|
||||
|
||||
def benchmark_embedded_python_api(path_to_model_file):
|
||||
""" Perform benchmark with dummy inputs, return inference requests's latency' mesurement result.
|
||||
:param path_to_model_file: if model is not provided, xml model file name.
|
||||
:return: latency metrics.
|
||||
"""
|
||||
def get_dummy_inputs(batch_size, input_info, requests):
|
||||
""" Generate dummpy inputs based on input and batch information.
|
||||
:param batch_size: batch size
|
||||
:param input_info: network's input infor
|
||||
:param requests: the network's requests
|
||||
:return: requests_input_data
|
||||
"""
|
||||
requests_input_data = []
|
||||
input_data = {}
|
||||
np_d_type = {'FP64': np.float64, 'I32': np.int32, 'FP32': np.float32, 'FP16': np.float16,
|
||||
'U16': np.uint16, 'I16': np.int16, 'U8': np.uint8, 'I8': np.int8}
|
||||
for key, value in input_info.items():
|
||||
m = []
|
||||
dt = np_d_type[value.precision]
|
||||
for x in value.input_data.shape:
|
||||
m.append(x)
|
||||
m[0] = m[0] * batch_size
|
||||
input_data[key] = np.empty(tuple(m), dtype=dt)
|
||||
for _ in range(len(requests)):
|
||||
requests_input_data.append(input_data)
|
||||
return requests_input_data
|
||||
|
||||
xml_filename = path_to_model_file
|
||||
bin_filename = path_to_model_file[:(len(path_to_model_file) - 4)] + '.bin'
|
||||
if not os.path.exists(bin_filename):
|
||||
logger.error('{} does not exist.'.format(bin_filename))
|
||||
return None
|
||||
|
||||
ie = IECore()
|
||||
ie_network = ie.read_network(xml_filename, bin_filename)
|
||||
device = 'CPU'
|
||||
config = {'PERF_COUNT': 'NO'}
|
||||
ie.set_config({'CPU_BIND_THREAD': str(benchmark_cfg['cpu_bind_thread'])}, device)
|
||||
if benchmark_cfg['nthreads'] is not None and benchmark_cfg['nthreads']:
|
||||
ie.set_config({'CPU_THREADS_NUM': str(benchmark_cfg['nthreads'])}, device)
|
||||
if benchmark_cfg['nstreams'] is not None:
|
||||
ie.set_config({'CPU_THROUGHPUT_STREAMS': str(benchmark_cfg['nstreams'])}, device)
|
||||
exe_network = ie.load_network(ie_network, device, config=config, num_requests=benchmark_cfg['nireq'])
|
||||
infer_requests = exe_network.requests
|
||||
batch_size = ie_network.batch_size
|
||||
request_queue = InferRequestsQueue(infer_requests)
|
||||
requests_input_data = get_dummy_inputs(batch_size, ie_network.input_info, infer_requests)
|
||||
infer_request = request_queue.get_idle_request()
|
||||
|
||||
# For warming up
|
||||
if benchmark_cfg['api_type'] == 'sync':
|
||||
infer_request.infer(requests_input_data[infer_request.id])
|
||||
else:
|
||||
infer_request.start_async(requests_input_data[infer_request.id])
|
||||
|
||||
request_queue.wait_all()
|
||||
request_queue.reset_times()
|
||||
start_time = datetime.now()
|
||||
exec_time = (datetime.now() - start_time).total_seconds()
|
||||
iteration = 0
|
||||
logger.info('Starting benchmark, will be done in {} seconds with {} api via python interface.'
|
||||
.format(benchmark_cfg['duration_seconds'], benchmark_cfg['api_type']))
|
||||
|
||||
while exec_time < benchmark_cfg['duration_seconds']:
|
||||
infer_request = request_queue.get_idle_request()
|
||||
if not infer_request:
|
||||
raise Exception('No idle Infer Requests!')
|
||||
if benchmark_cfg['api_type'] == 'sync':
|
||||
infer_request.infer(requests_input_data[infer_request.id])
|
||||
else:
|
||||
infer_request.start_async(requests_input_data[infer_request.id])
|
||||
iteration += 1
|
||||
exec_time = (datetime.now() - start_time).total_seconds()
|
||||
|
||||
request_queue.wait_all()
|
||||
t = np.array(request_queue.times)
|
||||
q75, q25 = np.percentile(t, [75, 25])
|
||||
IQR = q75 - q25
|
||||
filtered_times = t[t < (q75 + 1.5 * IQR)]
|
||||
logger.debug('benchmark result: latency_filtered_mean:{0:.3f}ms, latency_minum: {1:.3f}ms, \
|
||||
using {2} requests of total {3} ones for latency calcluation.'.format(
|
||||
filtered_times.mean(), filtered_times.min(), filtered_times.size, t.size))
|
||||
del exe_network
|
||||
del ie
|
||||
del ie_network
|
||||
return filtered_times.mean()
|
||||
|
||||
def benchmark_embedded_cpp_app(path_to_model_file):
|
||||
""" Perform benchmark with dummy inputs, return inference requests's latency' mesurement result.
|
||||
:param path_to_model_file: xml model file name
|
||||
:return: latency metrics, and performance counts for each layer if layer_awareness is Ture.
|
||||
"""
|
||||
|
||||
cmd_cb = [benchmark_cfg["benchmark_app_dir"], "-api", benchmark_cfg["api_type"], "-m",
|
||||
path_to_model_file]
|
||||
if benchmark_cfg["duration_seconds"] != 0:
|
||||
cmd_cb.append("-t")
|
||||
cmd_cb.append(str(benchmark_cfg["duration_seconds"]))
|
||||
if benchmark_cfg['nthreads'] is not None:
|
||||
cmd_cb.append("-nthreads")
|
||||
cmd_cb.append(str(benchmark_cfg['nthreads']))
|
||||
if benchmark_cfg['batch_size'] != 0:
|
||||
cmd_cb.append("-b")
|
||||
cmd_cb.append(str(benchmark_cfg['batch_size']))
|
||||
if benchmark_cfg['nstreams'] is not None:
|
||||
cmd_cb.append("-nstreams")
|
||||
cmd_cb.append(str(benchmark_cfg['nstreams']))
|
||||
cmd_cb.append("-pin")
|
||||
cmd_cb.append(str(benchmark_cfg['cpu_bind_thread']))
|
||||
if benchmark_cfg['nireq'] != 0:
|
||||
cmd_cb.append("-nireq")
|
||||
cmd_cb.append(str(benchmark_cfg['nireq']))
|
||||
|
||||
benchmark_finished = False
|
||||
logger.info(" ".join(cmd_cb))
|
||||
output = check_output(cmd_cb, shell=False)
|
||||
out = output.decode().split("\n")
|
||||
for line in out:
|
||||
if "Latency:" in line:
|
||||
latency_report = line.split()
|
||||
latency = float(latency_report[1])
|
||||
benchmark_finished = True
|
||||
if "Throughput:" in line:
|
||||
throughput_report = line.split()
|
||||
throughput = float(throughput_report[1])
|
||||
if not benchmark_finished:
|
||||
logger.error("Benchmark running fails.")
|
||||
logger.info("Benchmark result: latency: {0:.3f} ms, throughput: {1:.2f} FPS.".format(latency, throughput))
|
||||
return latency
|
||||
@@ -1,65 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import threading
|
||||
from ..utils.logger import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class InferReqWrap:
|
||||
def __init__(self, request, index, callback_queue):
|
||||
self.id = index
|
||||
self.request = request
|
||||
self.request.set_completion_callback(self.callback, self.id)
|
||||
self.callback_queue = callback_queue
|
||||
|
||||
def callback(self, statusCode, userdata):
|
||||
if userdata != self.id:
|
||||
logger.info('Request ID {} does not correspond to user data {}'.format(self.id, userdata))
|
||||
elif statusCode != 0:
|
||||
logger.info('Request {} failed with status code {}'.format(self.id, statusCode))
|
||||
self.callback_queue(self.id, self.request.latency)
|
||||
|
||||
def start_async(self, input_data):
|
||||
self.request.async_infer(input_data)
|
||||
|
||||
def infer(self, input_data):
|
||||
self.request.infer(input_data)
|
||||
self.callback_queue(self.id, self.request.latency)
|
||||
|
||||
|
||||
class InferRequestsQueue:
|
||||
def __init__(self, requests):
|
||||
self.idle_ids = []
|
||||
self.requests = []
|
||||
self.times = []
|
||||
inf_len = len(requests)
|
||||
for i in range(inf_len):
|
||||
self.requests.append(InferReqWrap(requests[i], i, self.put_idle_request))
|
||||
self.idle_ids.append(i)
|
||||
self.cv = threading.Condition()
|
||||
|
||||
def reset_times(self):
|
||||
self.times.clear()
|
||||
|
||||
def put_idle_request(self, i, latency):
|
||||
self.cv.acquire()
|
||||
self.times.append(latency)
|
||||
self.idle_ids.append(i)
|
||||
self.cv.notify()
|
||||
self.cv.release()
|
||||
|
||||
def get_idle_request(self):
|
||||
self.cv.acquire()
|
||||
while len(self.idle_ids) == 0:
|
||||
self.cv.wait()
|
||||
i = self.idle_ids.pop()
|
||||
self.cv.release()
|
||||
return self.requests[i]
|
||||
|
||||
def wait_all(self):
|
||||
self.cv.acquire()
|
||||
while len(self.idle_ids) != len(self.requests):
|
||||
self.cv.wait()
|
||||
self.cv.release()
|
||||
@@ -228,8 +228,6 @@ class Config(Dict):
|
||||
'opt_backend': None,
|
||||
},
|
||||
'TunableQuantization': {
|
||||
'tuning_scope': None,
|
||||
'estimator_tuning_scope': None,
|
||||
'outlier_prob_choices': None
|
||||
},
|
||||
'MagnitudeSparsity': {
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
@@ -1,24 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
|
||||
from ..api.engine import Engine
|
||||
from ..pipeline.pipeline import Pipeline
|
||||
|
||||
|
||||
class Optimizer(ABC):
|
||||
def __init__(self, config, pipeline: Pipeline, engine: Engine):
|
||||
""" Constructor
|
||||
:param config: optimizer config
|
||||
:param pipeline: pipeline of algorithms to optimize
|
||||
:param engine: entity responsible for communication with dataset
|
||||
"""
|
||||
self._config, self._pipeline, self._engine = config.params, pipeline, engine
|
||||
self.name = config.name
|
||||
|
||||
@abstractmethod
|
||||
def run(self, model):
|
||||
""" Run optimizer on model
|
||||
:param model: model to apply optimization
|
||||
"""
|
||||
@@ -1,6 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from openvino.tools.pot.utils.registry import Registry
|
||||
|
||||
OPTIMIZATION_ALGORITHMS = Registry('OptimizationAlgos')
|
||||
@@ -1,141 +0,0 @@
|
||||
# Tree-Structured Parzen Estimator (TPE) {#pot_compression_optimization_tpe_README}
|
||||
|
||||
## Overview
|
||||
Tree-Structured Parzen Estimator (TPE) algorithm is designed to optimize quantization hyperparameters to find quantization configuration that achieve an expected accuracy target and provide best possible latency improvement.
|
||||
TPE is an iterative process that uses history of evaluated hyperparameters to create probabilistic model, which is used to suggest next set of hyperparameters to evaluate.
|
||||
|
||||
Generally, the algorithm consists of the following steps:
|
||||
1. Define a domain of hyperparameter search space,
|
||||
2. Create an objective function which takes in hyperparameters and outputs a score (e.g., loss, root mean squared error, cross-entropy) that we want to minimize,
|
||||
3. Get couple of observations (score) using randomly selected set of hyperparameters,
|
||||
4. Sort the collected observations by score and divide them into two groups based on some quantile. The first group (x1) contains observations that gave the best scores and the second one (x2) - all other observations,
|
||||
5. Two densities l(x1) and g(x2) are modeled using Parzen Estimators (also known as kernel density estimators) which are a simple average of kernels centered on existing data points,
|
||||
6. Draw sample hyperparameters from l(x1), evaluating them in terms of l(x1)/g(x2), and returning the set that yields the minimum value under l(x1)/g(x1) corresponding to the greatest expected improvement. These hyperparameters are then evaluated on the objective function.
|
||||
7. Update the observation list from step 3
|
||||
8. Repeat step 4-7 with a fixed number of trials or until time limit is reached
|
||||
|
||||
TPE uses hyperparameters metadata returned by [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md)
|
||||
to create search space from which it chooses hyperparameters for algorithms.
|
||||
The configuration of hyperparameters metadata generation is done at [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md) level.
|
||||
For details, see description of the [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md).
|
||||
|
||||
For more information about TPE see [1].
|
||||
|
||||
> **NOTE**: TPE requires many iterations to converge to an optimal solution, and
|
||||
> it is recommended to run it for at least 200 iterations. Because every iteration
|
||||
> requires evaluation of a generated model, which means accuracy measurements on a
|
||||
> dataset and latency measurements using benchmark, this process may take from
|
||||
> 24 hours up to few days to complete, depending on a model.
|
||||
> Due to this, even though the TPE supports all OpenVINO™-supported models, it
|
||||
> is being continuously validated only on a subset of models:
|
||||
> * SSD MobileNet V1 COCO
|
||||
> * Mobilenet V2 1.0 224
|
||||
> * Faster R-CNN ResNet 50 COCO
|
||||
> * Faster R-CNN Inception V2 COCO
|
||||
> * YOLOv3 TF Full COCO
|
||||
|
||||
## Progress reporting
|
||||
|
||||
After every iteration TPE will output information about current iteration and best result.
|
||||
Current iteration result may have two forms:
|
||||
1. `INFO:compression.optimization.tpe.algorithm:Current iteration acc_loss: 0.0054 lat_diff: 1.2787 exec_time: 5.6135`
|
||||
2. `INFO:compression.optimization.tpe.algorithm:Params already evaluated. Returning previous result: acc_loss: 0.0054 lat_diff: 1.2787`
|
||||
|
||||
The first one is used when TPE evaluates new set of hyperparameters.
|
||||
The second one is used when TPE suggested set of hyperparameters that was already evaluated.
|
||||
In this case hyperparameters were not reevaluated, instead previous result for these hyperparameters was added to the observations list.
|
||||
|
||||
Best result is reported with following line:
|
||||
`INFO:compression.optimization.tpe.algorithm:Trial iteration end: 1 / 2 acc_loss: 0.0054 lat_diff: 1.2787`
|
||||
|
||||
In all of the above logs reported values are following:
|
||||
- `acc_loss` - relative accuracy loss, computed with formula `(FP32_accuracy - quantized_accuracy) / FP32_accuracy`, lower is better
|
||||
- `lat_diff` - latency difference compared to FP32 model, computed with formula `FP32_latency / quantized_latency`, higher is better
|
||||
- `exec_time` - execution time in minutes from start of the current run of the tool
|
||||
|
||||
## Parameters
|
||||
TPE parameters can be divided into two groups: mandatory and optional.
|
||||
|
||||
### Mandatory parameters
|
||||
- `"max_trials"` - maximum number of trails
|
||||
- `"trials_load_method"` - specifies whether to start from scratch or reuse previous results. It should be used in following manner:
|
||||
- `"cold_start"` - start trials from beginning. Logs from previous execution are removed
|
||||
- `"warm_start"` - continue execution using logs from previous execution up to the limit set by `"max_trials"` (may be larger than in previous execution). [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md) parameters impacting parameters metadata creation are ignored, because search space is retrieved from logs. May be used either after `"cold_start"` or `"fine_tune"`. If no previous logs exist then it behaves like `"cold_start"`
|
||||
- `"fine_tune"` - start new trials with new search space derived from best result achieved since last `"cold_start"`. [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md) is responsible for modifying parameter metadata to accommodate parameters used to get best result (for more details about parameter metadata generation see [TunableQuantization algorithm](../../algorithms/quantization/tunable_quantization/README.md)). If no previous logs exist then it behaves like `"cold_start"`
|
||||
- `"eval"` - load best result to get model
|
||||
- `"accuracy_loss"` - maximum acceptable relative accuracy loss in percentage
|
||||
- `"latency_reduce"` - target latency improvement versus original model
|
||||
- `"accuracy_weight"` and `"latency_weight"` - accuracy and latency weights used in loss function.
|
||||
These two parameters are intended to be set to 1.0, because accuracy and latency components in the loss function are designed to be balanced equally, so that the algorithm is able to achieve an expected accuracy target and provide best possible latency improvement.
|
||||
Changing `"accuracy_weight"`, which is left open for experimentation, is discouraged, but it is recommended to change `"latency_weight"` to 0 for configurations that do not change latency result, for example, when tuning parameters that only change numeric values of the parameters, such as quantization ranges, and do not change graph structure or data types
|
||||
- `"benchmark"` - latency measurement benchmark configuration. For details of configuration options see [Benchmark C++ Tool](https://docs.openvinotoolkit.org/latest/_inference_engine_samples_benchmark_app_README.html)
|
||||
|
||||
### Optional parameters
|
||||
- `"max_minutes"` - trials time limit. When it expires, the last trial is completed and the best result is returned
|
||||
- `"stop_on_target"` - flag to stop TPE trials when accuracy_loss and latency_reduce targets are reached.
|
||||
If false or not specified TPE will continue until max_trials or max_minutes is reached even if targets are reached earlier
|
||||
- `"eval_subset_size"` - subset of test data used to evaluate hyperparameters. The whole dataset is used if no parameter specified.
|
||||
- `"metrics"` - an optional list of reference metrics values.
|
||||
If not specified, all metrics will be calculated from the original model.
|
||||
It consists of tuples with the following parameters:
|
||||
- `"name"` - name of the metric to optimize
|
||||
- `"baseline_value"` - baseline metric value of the original model
|
||||
|
||||
Below is a fragment of the configuration file that shows overall structure of parameters for this algorithm.
|
||||
|
||||
```json
|
||||
/* Optimizer used to find "optimal" hyperparameters */
|
||||
"optimizer": {
|
||||
"name": "Tpe", // Optimizer name
|
||||
"params": {
|
||||
"max_trials": 100, // Maximum number of trails
|
||||
"max_minutes": 10, // [Optional] Trials time limit. When it expires, the last trial is completed and the best result is returned.
|
||||
"stop_on_target": true, // [Optional] Flag to stop TPE trials when accuracy_loss and latency_reduce targets are reached.
|
||||
// If false or not specified TPE will continue until max_trials or max_minutes is reached even if targets are reached earlier.
|
||||
"eval_subset_size": 2000, // [Optional] subset of test data used to evaluate hyperparameters. The whole dataset is used if no parameter specified.
|
||||
"trials_load_method": "cold_start", // Start from scratch or reuse previous results, supported options [cold_start, warm_start, fine_tune, eval]
|
||||
"accuracy_loss": 0.1, // Accuracy threshold (%)
|
||||
"latency_reduce": 1.5, // Target latency improvement versus original model
|
||||
"accuracy_weight": 1.0, // Accuracy weight in loss function
|
||||
"latency_weight": 1.0, // Latency weight in loss function
|
||||
// An optional list of reference metrics values.
|
||||
// If not specified, all metrics will be calculated from the original model.
|
||||
"metrics": [
|
||||
{
|
||||
"name": "accuracy", // Metric name
|
||||
"baseline_value": 0.72 // Baseline metric value of the original model
|
||||
}
|
||||
],
|
||||
"benchmark": {
|
||||
// Latency measurement benchmark configuration (https://docs.openvinotoolkit.org/latest/_inference_engine_samples_benchmark_app_README.html)
|
||||
"performance_count": false,
|
||||
"batch_size": 0,
|
||||
"nthreads": 4,
|
||||
"nstreams": 0,
|
||||
"nireq": 0,
|
||||
"api_type": "sync",
|
||||
"niter": 4,
|
||||
"duration_seconds": 30,
|
||||
"benchmark_app_dir": "<path to benchmark_app>" // Path to benchmark_app If not specified, Python base benchmark will be used. Use benchmark_app to reduce jitter in results.
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multiple node configuration
|
||||
Domain for hyperparameter search space can be vast. Searching best result is time-consuming process.
|
||||
The current implementation allow to use multiple machines to work together. For more information go to [Multi-node](multinode.md) description.
|
||||
|
||||
## Advantages
|
||||
1. TPE supports a wide variety of variables in parameter search space e.g., uniform, log-uniform, quantized log-uniform, normally-distributed real value, categorical.
|
||||
2. Less time to tune. Extremely computationally efficient than conventional methods.
|
||||
3. Scope to define the tuning time of quantization.
|
||||
4. Scope to define quantization search space and strategies from a wide range of OpenVINO™ quantization algorithms.
|
||||
5. Scope to define error tolerance and desired latency improvement. This algorithm guaranteed to get best possible accuracy and latency from optimal parameter combination (quantization algorithms and strategies).
|
||||
|
||||
## Drawbacks
|
||||
|
||||
TPE does not model interactions between hyperparameters.
|
||||
|
||||
## Reference
|
||||
[1] J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for Hyper-Parameter Optimization,” in Advances in Neural Information Processing Systems 24, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2011, pp. 2546–2554.
|
||||
@@ -1,2 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
@@ -1,544 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import os
|
||||
import time
|
||||
from copy import deepcopy
|
||||
from functools import partial
|
||||
from pathlib import Path
|
||||
|
||||
# pylint: disable=import-error
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import hyperopt as hpo
|
||||
from hyperopt import fmin, hp, STATUS_OK, Trials
|
||||
|
||||
from openvino.tools.pot.benchmark.benchmark import benchmark_embedded, set_benchmark_config
|
||||
from openvino.tools.pot.graph.model_utils import compress_model_weights
|
||||
from openvino.tools.pot.optimization.optimizer import Optimizer
|
||||
from openvino.tools.pot.optimization.tpe.multinode import Multinode
|
||||
from openvino.tools.pot.samplers.index_sampler import IndexSampler
|
||||
from openvino.tools.pot.utils.logger import get_logger
|
||||
from openvino.tools.pot.utils.object_dump import object_load, object_dump
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class TpeOptimizer(Optimizer):
|
||||
def __init__(self, config, pipeline, engine):
|
||||
super().__init__(config, pipeline, engine)
|
||||
self._search_space = None
|
||||
self._hpopt_trials = None
|
||||
self._hpopt_search_space = None
|
||||
self._evaluated_params = []
|
||||
self._tpe_params = {
|
||||
'n_initial_point': 10,
|
||||
'gamma': 0.3,
|
||||
'n_EI_candidates': 100,
|
||||
'prior_weight': 1.0
|
||||
}
|
||||
# Experimental option, currently not exposed to customer in external documentation
|
||||
if self._config.get('anneal', False):
|
||||
self._algo = hpo.anneal.suggest
|
||||
else:
|
||||
self._algo = None
|
||||
|
||||
self._fp32_acc = None
|
||||
self._fp32_lat = None
|
||||
self._engine.calculate_metrics = True
|
||||
self._base_metrics_list, _ = zip(*self._engine.get_metrics_attributes().items())
|
||||
self._opt_param = None
|
||||
self._best_result = {
|
||||
'best_loss': float('inf'),
|
||||
'best_acc_loss': float('inf'),
|
||||
'best_lat_diff': 0.0,
|
||||
'best_quantization_ratio': 0.0
|
||||
}
|
||||
|
||||
self.multinode = None
|
||||
self._start_time = None
|
||||
self._trial_load = self._config.get('trials_load_method', 'cold_start')
|
||||
self._debug = self._config.get('debug', False)
|
||||
|
||||
self._loss_function_config = self._prepare_loss_function_config(self._config)
|
||||
if 'benchmark' in self._config:
|
||||
set_benchmark_config(self._config.benchmark)
|
||||
|
||||
def _prepare_loss_function_config(self, config):
|
||||
return {
|
||||
'acc_th': config.get('accuracy_loss', 1) / 100,
|
||||
'lat_th': config.get('latency_reduce', 1),
|
||||
'quant_ratio_offset': config.get('expected_quantization_ratio', 0.5),
|
||||
'acc_weight': config.get('accuracy_weight', 1.0),
|
||||
'lat_weight': config.get('latency_weight', 1.0),
|
||||
'quant_ratio_weight': config.get('quantization_ratio_weight', 0.0)
|
||||
}
|
||||
|
||||
def _create_search_space(self, model, optimizer_state):
|
||||
param_grid = []
|
||||
for algo in self._pipeline.algo_seq:
|
||||
for param in algo.get_parameter_meta(model, optimizer_state):
|
||||
# Make param name unique among algorithms
|
||||
unique_param_name = algo.name + '_' + param[0]
|
||||
param_grid.append((unique_param_name, param[1], param[2]))
|
||||
return param_grid
|
||||
|
||||
def _configure_hpopt_search_space_and_params(self, search_space):
|
||||
self._hpopt_search_space = {}
|
||||
for param in search_space:
|
||||
if param[1] == 'choice':
|
||||
self._hpopt_search_space[param[0]] = hp.choice(param[0], param[2])
|
||||
else:
|
||||
raise ValueError('Unsupported param type: {}'.format(param[1]))
|
||||
# Find minimum number of choices for params with more than one choice
|
||||
multichoice_params = [len(param[2]) for param in search_space if param[1] == 'choice' and len(param[2]) > 1]
|
||||
min_param_size = min(multichoice_params) if multichoice_params else 1
|
||||
if not self._config.get('anneal', False):
|
||||
self._tpe_params['n_EI_candidates'] = min_param_size
|
||||
self._tpe_params['prior_weight'] = 1 / min_param_size
|
||||
self._algo = partial(hpo.tpe.suggest,
|
||||
n_startup_jobs=self._tpe_params['n_initial_point'],
|
||||
gamma=self._tpe_params['gamma'],
|
||||
n_EI_candidates=self._tpe_params['n_EI_candidates'],
|
||||
prior_weight=self._tpe_params['prior_weight'])
|
||||
|
||||
def generate_quantized_model(self, params, model):
|
||||
self._set_algorithms_params(params)
|
||||
return self._pipeline.run(deepcopy(model))
|
||||
|
||||
def _set_algorithms_params(self, params):
|
||||
for algo in self._pipeline.algo_seq:
|
||||
algo_params = {}
|
||||
for unique_param_name in [param for param in params.keys() if param.startswith(algo.name)]:
|
||||
param_name = unique_param_name[len(algo.name) + 1:]
|
||||
algo_params[param_name] = deepcopy(params[unique_param_name])
|
||||
algo.params = algo_params
|
||||
|
||||
def object_evaluation(self, params, model):
|
||||
if params in [ep['params'] for ep in self._evaluated_params]:
|
||||
previous_result = [ep for ep in self._evaluated_params if ep['params'] == params][0]
|
||||
previous_result['reevaluate_count'] += 1
|
||||
logger.info(
|
||||
'Params already evaluated. Returning previous result: acc_loss: {:.4f} lat_diff: {:.4f}'.format(
|
||||
previous_result['result']['acc_loss'], previous_result['result']['lat_diff']))
|
||||
return previous_result['result']
|
||||
if self._debug:
|
||||
object_dump(params, os.path.join(self._config.exec_log_dir, 'tpe_' + model.name + '_last_trial_params.p'))
|
||||
result = self._compute_metrics_for_params(params, model)
|
||||
self._update_result_with_calculated_loss(result)
|
||||
self._evaluated_params.append(
|
||||
{'params': params, 'result': result,
|
||||
'first_iteration': len(self._hpopt_trials.trials),
|
||||
'reevaluate_count': 0})
|
||||
return result
|
||||
|
||||
def _compute_metrics_for_params(self, params, model):
|
||||
qmodel = self.generate_quantized_model(params, model)
|
||||
lat = None
|
||||
#calculate benchmark on server
|
||||
if self.multinode is not None:
|
||||
if self._hpopt_trials is not None:
|
||||
lat = self.multinode.request_remote_benchmark(qmodel, len(self._hpopt_trials.trials))
|
||||
else:
|
||||
lat = self.multinode.request_remote_benchmark(qmodel, 0)
|
||||
# get int8 accuracy metric
|
||||
logger.info('Compute INT8 metrics')
|
||||
acc_new, lat_new = self._model_eval(qmodel, lat)
|
||||
quantization_params = [param for param in params.values() if 'quantize' in param]
|
||||
if not quantization_params:
|
||||
quantization_ratio = 1.0
|
||||
else:
|
||||
quantization_ratio = len([param for param in quantization_params if param['quantize'] == 1]) \
|
||||
/ len(quantization_params)
|
||||
quantization_params = []
|
||||
for algo in self._pipeline.algo_seq:
|
||||
for param in algo.params.values():
|
||||
if 'quantize' in param:
|
||||
quantization_params.append(param)
|
||||
if not quantization_params:
|
||||
real_quantization_ratio = 1.0
|
||||
else:
|
||||
real_quantization_ratio = len([param for param in quantization_params if param['quantize'] == 1]) \
|
||||
/ len(quantization_params)
|
||||
fp32_acc = self._fp32_acc[self._base_metrics_list[0]]
|
||||
int8_acc = acc_new[self._base_metrics_list[0]]
|
||||
acc_diff = (fp32_acc - int8_acc) / fp32_acc
|
||||
lat_diff = self._fp32_lat / lat_new
|
||||
return {
|
||||
'acc': acc_new,
|
||||
'lat': lat_new,
|
||||
'acc_loss': acc_diff,
|
||||
'lat_diff': lat_diff,
|
||||
'quantization_ratio': quantization_ratio,
|
||||
'real_quantization_ratio': real_quantization_ratio,
|
||||
'exec_time': (time.time() - self._start_time) / 60,
|
||||
'params': params,
|
||||
'status': STATUS_OK,
|
||||
'node': self.multinode.name if self.multinode is not None else "no_name"}
|
||||
|
||||
def _update_result_with_calculated_loss(self, result):
|
||||
result['loss'] = self.calculate_loss(result['acc_loss'], result['lat_diff'], result['real_quantization_ratio'],
|
||||
self._loss_function_config)
|
||||
logger.info('Current iteration acc_loss: {:.4f} lat_diff: {:.4f} exec_time: {:.4f}'.format(
|
||||
result['acc_loss'], result['lat_diff'], result['exec_time']))
|
||||
|
||||
@staticmethod
|
||||
def _calculate_acc_loss_component(acc_loss):
|
||||
return np.exp(acc_loss)
|
||||
|
||||
@staticmethod
|
||||
def _calculate_lat_diff_component(lat_diff):
|
||||
return np.log(np.power((1 / (1000 * lat_diff)), 8))
|
||||
|
||||
@staticmethod
|
||||
def _calculate_loss_function_scaling_components(acc_loss, lat_diff, config):
|
||||
acc_min = TpeOptimizer._calculate_acc_loss_component(0)
|
||||
acc_max = TpeOptimizer._calculate_acc_loss_component(acc_loss)
|
||||
if acc_max == acc_min:
|
||||
acc_max = TpeOptimizer._calculate_acc_loss_component(config['acc_th'])
|
||||
config['acc_min'] = acc_min
|
||||
config['acc_scale'] = 10 / np.abs(acc_max - acc_min)
|
||||
|
||||
lat_min = TpeOptimizer._calculate_lat_diff_component(lat_diff)
|
||||
lat_max = TpeOptimizer._calculate_lat_diff_component(1)
|
||||
if lat_min == lat_max:
|
||||
lat_min = TpeOptimizer._calculate_lat_diff_component(config['lat_th'])
|
||||
config['lat_min'] = lat_min
|
||||
config['lat_scale'] = 10 / np.abs(lat_max - lat_min)
|
||||
|
||||
@staticmethod
|
||||
def calculate_loss(acc_diff, lat_diff, quantization_ratio, config):
|
||||
gamma_penalty = 40 # penalty term
|
||||
quant_ratio_scaling_factor = 1.0 / min(1.0 - config['quant_ratio_offset'], config['quant_ratio_offset'])
|
||||
acc_loss_component = TpeOptimizer._calculate_acc_loss_component(acc_diff)
|
||||
lat_loss_component = TpeOptimizer._calculate_lat_diff_component(lat_diff)
|
||||
acc_weight = config['acc_weight'] if acc_diff > config['acc_th'] else 0.0
|
||||
if acc_weight == 0 and config['lat_weight'] == 0 and config['quant_ratio_weight'] == 0:
|
||||
acc_weight = 1.0
|
||||
loss = acc_weight * (config['acc_scale'] * (acc_loss_component - config['acc_min'])) \
|
||||
+ config['lat_weight'] * (config['lat_scale'] * (lat_loss_component - config['lat_min'])) \
|
||||
- config['quant_ratio_weight'] * 5 \
|
||||
* np.tanh(3 * quant_ratio_scaling_factor * (quantization_ratio - config['quant_ratio_offset']))
|
||||
if acc_diff > config['acc_th']:
|
||||
loss += 2 * gamma_penalty
|
||||
elif lat_diff < config['lat_th']:
|
||||
loss += gamma_penalty
|
||||
return loss
|
||||
|
||||
def _model_eval(self, model, lat):
|
||||
if not self._config.keep_uncompressed_weights:
|
||||
compress_model_weights(model)
|
||||
self._engine.set_model(model)
|
||||
subset_indices = range(self._config.eval_subset_size) \
|
||||
if self._config.eval_subset_size else range(len(self._engine.data_loader))
|
||||
acc, _ = self._engine.predict(sampler=IndexSampler(subset_indices=subset_indices),
|
||||
print_progress=self._debug)
|
||||
if lat is None:
|
||||
lat = benchmark_embedded(model)
|
||||
return acc, lat
|
||||
|
||||
def _update_best_result(self, best_result_file):
|
||||
if not self._hpopt_trials:
|
||||
raise Exception(
|
||||
'No trials loaded to get best result')
|
||||
trials_results = pd.DataFrame(self._hpopt_trials.results)
|
||||
|
||||
if not trials_results[trials_results.acc_loss <= self._loss_function_config['acc_th']].empty:
|
||||
# If accuracy threshold reached, choose best latency
|
||||
best_result = trials_results[trials_results.acc_loss <= self._loss_function_config['acc_th']] \
|
||||
.reset_index(drop=True).sort_values(by=['lat_diff', 'acc_loss'], ascending=[False, True]) \
|
||||
.reset_index(drop=True).loc[0]
|
||||
else:
|
||||
# If accuracy threshold is not reached, choose based on loss function
|
||||
best_result = trials_results.sort_values('loss', ascending=True).reset_index(drop=True).loc[0]
|
||||
|
||||
update_best_result = False
|
||||
if not self._best_result['best_loss']:
|
||||
update_best_result = True
|
||||
elif self._best_result['best_acc_loss'] <= self._loss_function_config['acc_th']:
|
||||
if best_result['acc_loss'] <= self._loss_function_config['acc_th'] \
|
||||
and best_result['lat_diff'] > self._best_result['best_lat_diff']:
|
||||
update_best_result = True
|
||||
else:
|
||||
if best_result['acc_loss'] <= self._loss_function_config['acc_th'] or \
|
||||
best_result['loss'] < self._best_result['best_loss']:
|
||||
update_best_result = True
|
||||
|
||||
if update_best_result:
|
||||
object_dump(dict(best_result), best_result_file)
|
||||
best_result.to_csv(best_result_file + '.csv', header=False)
|
||||
self._opt_param = best_result['params']
|
||||
self._best_result['best_loss'] = best_result['loss']
|
||||
self._best_result['best_acc_loss'] = best_result['acc_loss']
|
||||
self._best_result['best_lat_diff'] = best_result['lat_diff']
|
||||
self._best_result['best_quantization_ratio'] = best_result['real_quantization_ratio']
|
||||
|
||||
logger.info('Trial iteration end: {} / {} acc_loss: {:.4f} lat_diff: {:.4f} '.format(
|
||||
len(self._hpopt_trials.trials), self._config.max_trials, self._best_result['best_acc_loss'],
|
||||
self._best_result['best_lat_diff']))
|
||||
|
||||
def _restore_best_result(self, best_result_file):
|
||||
""" restore trial database and state"""
|
||||
best_result = object_load(best_result_file)
|
||||
self._opt_param = best_result['params']
|
||||
self._best_result['best_loss'] = best_result['loss']
|
||||
self._best_result['best_acc_loss'] = best_result['acc_loss']
|
||||
self._best_result['best_lat_diff'] = best_result['lat_diff']
|
||||
self._best_result['best_quantization_ratio'] = best_result['real_quantization_ratio']
|
||||
|
||||
def _create_tuned_loss_config(self, loss_config_file, model):
|
||||
search_space = self._create_search_space(model, {'first_iteration': False, 'fully_quantized': True})
|
||||
params = {}
|
||||
for param in search_space:
|
||||
if param[1] == 'choice':
|
||||
params[param[0]] = param[2][0]
|
||||
else:
|
||||
raise ValueError('Unsupported param type: {}'.format(param[1]))
|
||||
result = self._compute_metrics_for_params(params, model)
|
||||
TpeOptimizer._calculate_loss_function_scaling_components(result['acc_loss'], result['lat_diff'],
|
||||
self._loss_function_config)
|
||||
self._update_result_with_calculated_loss(result)
|
||||
self._evaluated_params.append(
|
||||
{'params': params, 'result': result,
|
||||
'first_iteration': 0,
|
||||
'reevaluate_count': 0})
|
||||
self._set_algorithms_params({})
|
||||
object_dump(self._loss_function_config, loss_config_file)
|
||||
|
||||
def _restore_tuned_loss_config(self, loss_config_file):
|
||||
""" restore loss function configuration"""
|
||||
loss_config = object_load(loss_config_file)
|
||||
self._loss_function_config['acc_min'] = loss_config['acc_min']
|
||||
self._loss_function_config['acc_scale'] = loss_config['acc_scale']
|
||||
self._loss_function_config['lat_min'] = loss_config['lat_min']
|
||||
self._loss_function_config['lat_scale'] = loss_config['lat_scale']
|
||||
|
||||
def _run_trials(self, max_trials, max_minutes, trials_file, best_result_file, model):
|
||||
trials_count = len(self._hpopt_trials.trials)
|
||||
while trials_count < max_trials:
|
||||
if max_minutes != 0 and (time.time() - self._start_time) > (max_minutes * 60):
|
||||
logger.info('Time limit of {} minutes reached'.format(max_minutes))
|
||||
break
|
||||
trials_count += 1
|
||||
logger.info('Trial iteration start: {} / {}'.format(trials_count, max_trials))
|
||||
fmin(partial(self.object_evaluation, model=model),
|
||||
space=self._hpopt_search_space,
|
||||
algo=self._algo,
|
||||
max_evals=trials_count,
|
||||
trials=self._hpopt_trials,
|
||||
show_progressbar=False)
|
||||
self._save_trials(trials_file)
|
||||
self._update_remote_trials()
|
||||
trials_count = len(self._hpopt_trials.trials)
|
||||
self._update_best_result(best_result_file)
|
||||
if self._config.get('stop_on_target', False) \
|
||||
and self._best_result['best_acc_loss'] <= self._loss_function_config['acc_th'] \
|
||||
and self._best_result['best_lat_diff'] >= self._loss_function_config['lat_th']:
|
||||
logger.info('Accuracy and latency reached')
|
||||
break
|
||||
|
||||
def _get_fp32_metrics(self, model, fp32_metric_file):
|
||||
# get fp32 metrics from .p file
|
||||
if Path(fp32_metric_file).exists():
|
||||
logger.info('load fp32 metrics directly from {}'.format(fp32_metric_file))
|
||||
self._fp32_acc, self._fp32_lat = object_load(fp32_metric_file)
|
||||
else:
|
||||
# get metrics from config .json file if exist
|
||||
self._fp32_acc = {metric.name: metric.baseline_value
|
||||
for metric in self._config.metrics
|
||||
if metric.name in self._base_metrics_list} \
|
||||
if self._config.metrics and \
|
||||
all('baseline_value' in metric.keys() for metric in self._config.metrics) \
|
||||
else None
|
||||
if self._fp32_acc is not None:
|
||||
logger.info('load fp32_acc metric directly from config file')
|
||||
# if fp32_acc alredy loaded from config run benchmark only for fp32_lat
|
||||
logger.info('load fp32_lat metric from benchmark')
|
||||
self._fp32_lat = benchmark_embedded(model)
|
||||
# if can't find metrics in config file nor in .p file run evaluation
|
||||
else:
|
||||
logger.info('compute fp32 metrics once and save to {}'.format(fp32_metric_file))
|
||||
self._fp32_acc, self._fp32_lat = self._model_eval(model, None)
|
||||
object_dump((self._fp32_acc, self._fp32_lat), fp32_metric_file)
|
||||
pd.Series({'acc': self._fp32_acc, 'lat': self._fp32_lat}).to_csv(fp32_metric_file + '.csv',
|
||||
header=False)
|
||||
if self.multinode is not None:
|
||||
self._fp32_lat = self.multinode.update_or_restore_fp32(self._fp32_acc, self._fp32_lat)
|
||||
|
||||
def _get_tuned_loss_config(self, model, loss_config_file):
|
||||
if Path(loss_config_file).exists():
|
||||
logger.info('load loss function config directly from {}'.format(loss_config_file))
|
||||
self._restore_tuned_loss_config(loss_config_file)
|
||||
else:
|
||||
logger.info('compute loss function config and save in {}'.format(loss_config_file))
|
||||
self._create_tuned_loss_config(loss_config_file, model)
|
||||
|
||||
def _update_remote_trials(self):
|
||||
if self.multinode is not None:
|
||||
self._hpopt_trials = self.multinode.update_remote_trials(self._hpopt_trials)
|
||||
self._evaluated_params = self.multinode.update_remote_evaluated_params(self._evaluated_params)
|
||||
|
||||
def _restore_remote_trials(self):
|
||||
self._hpopt_trials = self.multinode.restore_remote_trials()
|
||||
self._evaluated_params = self.multinode.restore_remote_evaluated_params()
|
||||
|
||||
def _update_remote_startup_data(self):
|
||||
if self.multinode is not None:
|
||||
self.multinode.update_remote_search_space(self._search_space)
|
||||
self._update_remote_trials()
|
||||
|
||||
def _compute_first_iteration_and_create_final_search_space(self, model, tpe_config_file, best_result_file,
|
||||
trials_file):
|
||||
self._hpopt_trials = Trials()
|
||||
# Run first iteration
|
||||
self._search_space = self._create_search_space(model,
|
||||
{'first_iteration': True, 'fully_quantized': False})
|
||||
self._configure_hpopt_search_space_and_params(self._search_space)
|
||||
logger.info('Trial iteration start: {} / {}'.format(1, self._config.max_trials))
|
||||
fmin(partial(self.object_evaluation, model=model),
|
||||
space=self._hpopt_search_space,
|
||||
algo=self._algo,
|
||||
max_evals=1,
|
||||
trials=self._hpopt_trials,
|
||||
show_progressbar=False)
|
||||
self._update_best_result(best_result_file)
|
||||
|
||||
# Generate final search space and TPE config
|
||||
self._search_space = self._create_search_space(model,
|
||||
{'first_iteration': False, 'fully_quantized': False})
|
||||
self._configure_hpopt_search_space_and_params(self._search_space)
|
||||
self._update_remote_startup_data()
|
||||
# Remove trials file before updating config and trials to prevent mismatch between config and trials files
|
||||
if Path(trials_file).exists():
|
||||
os.remove(trials_file)
|
||||
self._save_tpe_config(tpe_config_file)
|
||||
self._save_trials(trials_file)
|
||||
|
||||
def start_trials(self, model, fp32_metric_file, tpe_config_file, trials_file, best_result_file, loss_config_file):
|
||||
self._start_time = time.time()
|
||||
self._get_fp32_metrics(model, fp32_metric_file)
|
||||
self._get_tuned_loss_config(model, loss_config_file)
|
||||
|
||||
if self.multinode is not None and self.multinode.type == 'client':
|
||||
self._search_space = self.multinode.restore_remote_search_space()
|
||||
self._configure_hpopt_search_space_and_params(self._search_space)
|
||||
self._restore_remote_trials()
|
||||
|
||||
elif self._trial_load == 'warm_start':
|
||||
self._restore_tpe_config(tpe_config_file)
|
||||
self._restore_trials(trials_file)
|
||||
# If trials were restored upload this data to remote database
|
||||
self._update_remote_startup_data()
|
||||
logger.info('Rerunning from {} trials to {} trials'
|
||||
.format(len(self._hpopt_trials.trials), self._config.max_trials))
|
||||
|
||||
elif self._trial_load == 'cold_start':
|
||||
self._compute_first_iteration_and_create_final_search_space(model, tpe_config_file, best_result_file,
|
||||
trials_file)
|
||||
|
||||
elif self._trial_load == 'fine_tune':
|
||||
self._restore_best_result(best_result_file)
|
||||
logger.info('Found best result! acc_loss: {:.4f} lat_diff: {:.4f} '.format(
|
||||
self._best_result['best_acc_loss'], self._best_result['best_lat_diff']))
|
||||
self._set_algorithms_params(self._opt_param)
|
||||
# Remove best loss to "reset" best_loss in _update_best_result
|
||||
self._best_result['best_loss'] = None
|
||||
self._compute_first_iteration_and_create_final_search_space(model, tpe_config_file, best_result_file,
|
||||
trials_file)
|
||||
|
||||
if self.multinode is not None and self.multinode.type == 'server' and self.multinode.mode == 'master':
|
||||
self.multinode.calculate_remote_requests()
|
||||
self._restore_remote_trials()
|
||||
self._update_best_result(best_result_file)
|
||||
else:
|
||||
self._run_trials(self._config.max_trials,
|
||||
self._config.max_minutes if self._config.max_minutes else 0,
|
||||
trials_file, best_result_file, model)
|
||||
|
||||
if self._best_result['best_acc_loss'] <= self._loss_function_config['acc_th'] \
|
||||
and self._best_result['best_lat_diff'] >= self._loss_function_config['lat_th']:
|
||||
logger.info('Congratulation parameters found to meet latency and accuracy criteria')
|
||||
elif self._best_result['best_acc_loss'] <= self._loss_function_config['acc_th']:
|
||||
logger.info('TPE satisfy accuracy criteria but can not satisfy latency criteria')
|
||||
else:
|
||||
logger.info('Sorry, TPE can not satisfy accuracy and latency criteria')
|
||||
|
||||
def _save_tpe_config(self, tpe_config_file):
|
||||
tpe_config_object = self._search_space
|
||||
object_dump(tpe_config_object, tpe_config_file)
|
||||
|
||||
def _restore_tpe_config(self, tpe_config_file):
|
||||
self._search_space = object_load(tpe_config_file)
|
||||
self._configure_hpopt_search_space_and_params(self._search_space)
|
||||
|
||||
def _save_trials(self, trials_log):
|
||||
""" save trial result to log file"""
|
||||
trials_object = (self._hpopt_trials, self._evaluated_params)
|
||||
object_dump(trials_object, trials_log)
|
||||
tpe_trials_results = pd.DataFrame(self._hpopt_trials.results)
|
||||
csv_file = trials_log + '.csv'
|
||||
tpe_trials_results.to_csv(csv_file)
|
||||
tpe_trials_results = pd.DataFrame(self._evaluated_params)
|
||||
csv_file = trials_log + '.params.csv'
|
||||
tpe_trials_results.to_csv(csv_file)
|
||||
|
||||
def _restore_trials(self, trials_log):
|
||||
""" restore trial database and state"""
|
||||
trials_object = object_load(trials_log)
|
||||
|
||||
self._hpopt_trials = trials_object[0]
|
||||
self._evaluated_params = trials_object[1]
|
||||
|
||||
def run(self, model):
|
||||
""" this function applies TPE algorithm
|
||||
:param model: model to apply algo
|
||||
:return model with inserted and filled FakeQuantize nodes
|
||||
"""
|
||||
if self._config.eval_subset_size:
|
||||
fp32_metric_file = 'tpe_' + model.name + '_fp32_metric_' + str(self._config.eval_subset_size) + '.p'
|
||||
else:
|
||||
fp32_metric_file = 'tpe_' + model.name + '_fp32_metric.p'
|
||||
fp32_metric_file = os.path.join(self._config.model_log_dir, fp32_metric_file)
|
||||
trials_file = os.path.join(self._config.model_log_dir, 'tpe_' + model.name + '_trials.p')
|
||||
tpe_config_file = os.path.join(self._config.model_log_dir, 'tpe_' + model.name + '_config.p')
|
||||
best_result_file = os.path.join(self._config.model_log_dir, 'tpe_' + model.name + '_best_result.p')
|
||||
loss_config_file = os.path.join(self._config.model_log_dir, 'tpe_' + model.name + '_loss_config.p')
|
||||
|
||||
if 'multinode' in self._config:
|
||||
self.multinode = Multinode(self._config, model.name)
|
||||
config = self.multinode.update_or_restore_config(self._config)
|
||||
if self.multinode.type == "client":
|
||||
self._loss_function_config = self._prepare_loss_function_config(config)
|
||||
self._config.max_trials = config.get('max_trials', None)
|
||||
self._config.max_minutes = config.get('max_minutes', None)
|
||||
self._config.eval_subset_size = config.get('eval_subset_size', None)
|
||||
set_benchmark_config(config['benchmark'])
|
||||
|
||||
# Remove any old data on cold_start
|
||||
if self._trial_load == 'cold_start':
|
||||
if Path(fp32_metric_file).exists():
|
||||
os.remove(fp32_metric_file)
|
||||
if Path(trials_file).exists():
|
||||
os.remove(trials_file)
|
||||
if Path(tpe_config_file).exists():
|
||||
os.remove(tpe_config_file)
|
||||
if Path(best_result_file).exists():
|
||||
os.remove(best_result_file)
|
||||
if Path(loss_config_file).exists():
|
||||
os.remove(loss_config_file)
|
||||
|
||||
if self._trial_load != 'eval':
|
||||
self.start_trials(model, fp32_metric_file, tpe_config_file, trials_file, best_result_file, loss_config_file)
|
||||
if self.multinode is not None:
|
||||
self.multinode.cleanup()
|
||||
else:
|
||||
# restore best result for eval
|
||||
if Path(best_result_file).exists():
|
||||
self._restore_best_result(best_result_file)
|
||||
else:
|
||||
raise Exception(
|
||||
'WARNING: Best result file {} does not exist '.format(best_result_file))
|
||||
logger.info(' -Generate final INT8 model now')
|
||||
final_model = self.generate_quantized_model(self._opt_param, model)
|
||||
|
||||
return final_model
|
||||
@@ -1,32 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
try:
|
||||
# pylint: disable=unused-import
|
||||
import hyperopt
|
||||
from .algorithm import TpeOptimizer
|
||||
|
||||
HYPEROPT_AVAILABLE = True
|
||||
except ImportError:
|
||||
HYPEROPT_AVAILABLE = False
|
||||
|
||||
|
||||
from openvino.tools.pot.optimization.optimizer import Optimizer
|
||||
from openvino.tools.pot.optimization.optimizer_selector import OPTIMIZATION_ALGORITHMS
|
||||
|
||||
|
||||
@OPTIMIZATION_ALGORITHMS.register('Tpe')
|
||||
class Tpe(Optimizer):
|
||||
def __init__(self, config, pipeline, engine):
|
||||
super().__init__(config, pipeline, engine)
|
||||
if HYPEROPT_AVAILABLE:
|
||||
self.optimizer = TpeOptimizer(config, pipeline, engine)
|
||||
else:
|
||||
raise ModuleNotFoundError(
|
||||
'Cannot import the hyperopt package which is a dependency '
|
||||
'of the TPE algorithm. '
|
||||
'Please install hyperopt via `pip install hyperopt==0.1.2 pandas==0.24.2`'
|
||||
)
|
||||
|
||||
def run(self, model):
|
||||
return self.optimizer.run(model)
|
||||
@@ -1,111 +0,0 @@
|
||||
# TPE Multiple Node Configuration Based on MongoDB Database. {#pot_compression_optimization_tpe_multinode}
|
||||
|
||||
The multi-node configuration purpose is to reduce the execution time of TPE algorithm.
|
||||
The main execution model of multi-node configuration is to run the identical script
|
||||
on many machines and share trials, loss function configuration,
|
||||
evaluated parameters and search space objects between them.
|
||||
|
||||
Multi-node configuration can work with two modes: master and peer.
|
||||
In peer mode, all machines calculate latency for evaluated models themselves.
|
||||
It means that for this configuration nodes used need to be homogeneous.
|
||||
In master configuration, only one machine called server calculates latency
|
||||
for evaluated models. Other machines are called clients and are responsible for generating the model based on parameters
|
||||
chosen by TPE, evaluating its accuracy and sending it to the server node
|
||||
for latency measurement. This approach enables precise latency measurements in the environment with
|
||||
machines with different hardware as long as server machine correctly represent target hardware.
|
||||
|
||||
The final result can be taken from any of the nodes.
|
||||
|
||||
## Configuration of nodes
|
||||
In this configuration, we distinguish two types of nodes. One is server
|
||||
which is responsible for proper "search space" and "loss function config" creation
|
||||
and client nodes which read data produced by server and use it for further
|
||||
best result search and evaluation.
|
||||
Server and Client .json config file changes:
|
||||
```json
|
||||
"optimizer": {
|
||||
"name": "Tpe",
|
||||
"params": {
|
||||
"multinode": {
|
||||
"name": "node_name", ← optional
|
||||
"type": "server", ← for server node
|
||||
"type": "client", ← for client node
|
||||
"server_addr": "<server_ip_addr>:<server_port_number>",
|
||||
"tag": "group_name", ← optional
|
||||
"mode": "peer"← optional
|
||||
},
|
||||
"max_trials": 10,
|
||||
"trials_load_method": "cold_start",
|
||||
...,
|
||||
}
|
||||
}
|
||||
```
|
||||
`parameters:`
|
||||
* `"name"`: Name saved in trials.csv file, mainly for debug purpose,
|
||||
* `"type"`: Can be "server" or "client"
|
||||
* `"server_addr"`: <server_ip_addr>: IP address of the machine where MongoDB database
|
||||
is configured. It can be different from any Node IP used for TPE execution.
|
||||
<server_port_number>: Port number of MongoDB database, by default it's 27017
|
||||
* `"tag"`: Name for a group of systems working together. Without this tag,
|
||||
systems will be grouped by the model they are working on.
|
||||
If more than one group of systems is working on the same model using
|
||||
the same MongoDB database, their results will collide.
|
||||
* `"mode"`: "peer" or "master" mode selection.
|
||||
|
||||
## How to run TPE in multi node configuration
|
||||
For every node, you need to have an environment prepared in the same way as for the regular run.
|
||||
Models and Datasets should be prepared in configuration files.
|
||||
When you add "multinode" parameter to the configuration file and MongoDB is active and running
|
||||
you need to run a tool on every node you want to be part of the searching group of machines.
|
||||
|
||||
The server should be run first because it needs to prepare data for other nodes.
|
||||
When the server node starts its 2nd Trial clients will start their search.
|
||||
|
||||
Steps needed to run multi-node configuration:
|
||||
|
||||
1. Add to your base configuration file 'multinode' parameters with one server and client
|
||||
type for rest of nodes,
|
||||
2. Create 'pot' database in MongoDB instance,
|
||||
3. Run server node as first (*server will perform cleanup on database*),
|
||||
4. Run client nodes,
|
||||
|
||||
## How to select mode
|
||||
When a mode parameter is set to peer all nodes (clients and server) will search for the best
|
||||
result and evaluate models (for accuracy and latency). In this mode nodes should be homogeneous,
|
||||
so that result of the benchmark on the same model would be similar on every node.
|
||||
This allows calculating correct loss and achieves better and faster convergence.
|
||||
|
||||
The Master mode is more reliable for latency calculation. Only the server node calculates
|
||||
latency for the rest of the nodes, but it is not doing accuracy evaluation
|
||||
and does not take part in searching for the best result. That's why this mode is slightly
|
||||
slower than the previous one.
|
||||
|
||||
Select master mode when:
|
||||
* machines with different hardware configuration are used (memory, CPU),
|
||||
* layer option is set in configuration file, (latency sensitive),
|
||||
* latency is main factor to be improved,
|
||||
* for number of nodes 4+.
|
||||
|
||||
Select peer mode when:
|
||||
* machines used are homogeneous,
|
||||
* range estimator option is set in configuration file (accuracy sensitive),
|
||||
* accuracy is main factor to be improved,
|
||||
* for limited number of nodes 1-3.
|
||||
|
||||
## How it works
|
||||
All synchronization is done by the MongoDB database. There is no direct communication between servers and clients.
|
||||
The client needs to wait for information about loss function configuration, fp32 metrics, or search space until
|
||||
the server push this data to the database.
|
||||
|
||||
## Results
|
||||
Time in minutes for TPE execution for 200 trials on ssd-mobilenetv1 and COCO dataset.
|
||||
|
||||
No. of nodes | master mode | peer mode
|
||||
----------------- | ------ | -----
|
||||
1| n/a | 865
|
||||
2| n/a | 488
|
||||
3| 510 | 329
|
||||
4| 325 | 250
|
||||
5| 260 | 200
|
||||
6| 208 | 171
|
||||
7| 188 | 153
|
||||
@@ -1,473 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import time
|
||||
from tempfile import gettempdir
|
||||
|
||||
# pylint: disable=import-error
|
||||
from hyperopt import trials_from_docs
|
||||
|
||||
from openvino.tools.pot.benchmark.benchmark import benchmark_embedded
|
||||
from openvino.tools.pot.graph import save_model
|
||||
from openvino.tools.pot.utils.logger import get_logger
|
||||
from openvino.tools.pot.utils.object_dump import object_dumps, object_loads
|
||||
from openvino.tools.pot.utils.utils import create_tmp_dir
|
||||
|
||||
try:
|
||||
from pymongo import MongoClient
|
||||
from pymongo.errors import ServerSelectionTimeoutError
|
||||
from bson.objectid import ObjectId
|
||||
import gridfs
|
||||
except ImportError:
|
||||
raise ImportError('Pymongo is not installed. Please install it before using multinode configuration.')
|
||||
|
||||
REMOTE_DATA_INFO_FREQ_S = 10
|
||||
RESTORATION_TIME_LIMIT_S = 60 * 60
|
||||
TRIALS_RESTORATION_TIME_LIMIT_S = 10
|
||||
UNLOCK_TIME_LIMIT_S = 60
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class Multinode:
|
||||
def __init__(self, system_config, config_name):
|
||||
self.type = None
|
||||
self.server_addr = None
|
||||
self.tag = config_name
|
||||
self.name = 'no_name'
|
||||
self.time_limit = RESTORATION_TIME_LIMIT_S
|
||||
self.unlock_time_limit_s = UNLOCK_TIME_LIMIT_S
|
||||
self.config = system_config.multinode
|
||||
self.client = None
|
||||
self.trials = None
|
||||
self.model = None
|
||||
self.search_space = None
|
||||
self.evaluated_params = None
|
||||
self.params = None
|
||||
self.remote_fs = None
|
||||
self.mode = 'peer'
|
||||
self.id = None
|
||||
self.wait_for_client = True
|
||||
self._set_config()
|
||||
|
||||
def _set_config(self):
|
||||
if all(key in self.config for key in ['type', 'server_addr']):
|
||||
self.type = self.config['type']
|
||||
if self.type not in ['server', 'client']:
|
||||
raise Exception('Illegal value for type in multinode config!')
|
||||
self.server_addr = self.config["server_addr"]
|
||||
else:
|
||||
raise Exception('Missing "server_addr" or "type" in multinode config')
|
||||
if 'mode' in self.config:
|
||||
if self.config['mode'] in ['master', 'peer']:
|
||||
self.mode = self.config['mode']
|
||||
if 'tag' in self.config:
|
||||
self.tag = self.config['tag']
|
||||
if 'name' in self.config:
|
||||
self.name = self.config['name']
|
||||
if 'time_limit' in self.config:
|
||||
self.time_limit = self.config['time_limit']
|
||||
self.client = MongoClient(self.server_addr)
|
||||
database = self.client['pot']
|
||||
self.trials = database[self.tag + '.trials']
|
||||
self.model = database[self.tag + '.model']
|
||||
self.search_space = database[self.tag + '.search_space']
|
||||
self.evaluated_params = database[self.tag + '.evaluated_params']
|
||||
self.params = database[self.tag + '.params']
|
||||
self.fp32 = database[self.tag + '.fp32']
|
||||
self.clients = database[self.tag + '.clients']
|
||||
self._init_gridfs(database)
|
||||
self._clear_remote_data(database)
|
||||
|
||||
def _init_gridfs(self, database):
|
||||
""" Initialize gridfs for mongodb."""
|
||||
try:
|
||||
self.client.server_info()
|
||||
except ServerSelectionTimeoutError:
|
||||
raise Exception('WARNING: Could not connect to MongoDB!!!')
|
||||
# GridFS
|
||||
self.remote_fs = gridfs.GridFSBucket(database, bucket_name=self.tag)
|
||||
|
||||
def _clear_remote_data(self, database):
|
||||
""" Removes remote data from remote database """
|
||||
chunks = database[self.tag + '.chunks']
|
||||
files = database[self.tag + '.files']
|
||||
if self.type == 'server':
|
||||
if self.trials.count_documents({}):
|
||||
self.trials.drop()
|
||||
logger.info('Remote trials found and removed.')
|
||||
if self.search_space.count_documents({}):
|
||||
self.search_space.drop()
|
||||
logger.info('Remote search space found and removed.')
|
||||
if self.evaluated_params.count_documents({}):
|
||||
self.evaluated_params.drop()
|
||||
logger.info('Remote evaluated parameters set found and removed.')
|
||||
if self.params.count_documents({}):
|
||||
self.params.drop()
|
||||
logger.info('Remote config found and removed.')
|
||||
# GridFS data cleanup
|
||||
if chunks.count_documents({}) or files.count_documents({}):
|
||||
chunks.drop()
|
||||
files.drop()
|
||||
logger.info('Remote trials file found and removed.')
|
||||
if self.model.count_documents({}):
|
||||
self.model.drop()
|
||||
logger.info('Remote model params found and removed.')
|
||||
if self.clients.count_documents({}):
|
||||
self.clients.drop()
|
||||
logger.info('Remote clients found and removed.')
|
||||
if self.fp32.count_documents({}):
|
||||
self.fp32.drop()
|
||||
logger.info('Remote fp32 data found and removed.')
|
||||
|
||||
def update_or_restore_config(self, _config, valid=True):
|
||||
""" Update or restore remote loss function config."""
|
||||
if self.type == 'client':
|
||||
_config = self._restore_remote_config()
|
||||
logger.info('Remote config restored')
|
||||
else:
|
||||
self._update_remote_config(_config, valid)
|
||||
logger.info('Remote config updated')
|
||||
return _config
|
||||
|
||||
def update_or_restore_fp32(self, _fp32_acc, _fp32_lat):
|
||||
""" Update or restore fp32 data (lat only)."""
|
||||
if self.mode == 'master':
|
||||
if self.type == 'server':
|
||||
self._update_remote_fp32(_fp32_acc, _fp32_lat)
|
||||
if self.type == 'client':
|
||||
_, lat = self._restore_fp32()
|
||||
return lat
|
||||
return _fp32_lat
|
||||
|
||||
def _update_remote_fp32(self, _fp32_acc, _fp32_lat):
|
||||
""" Update remote fp32 function config."""
|
||||
if self.type == 'server':
|
||||
self.fp32.insert_one({
|
||||
'fp32_lat': object_dumps(_fp32_lat),
|
||||
'fp32_acc': object_dumps(_fp32_acc)})
|
||||
logger.info('Remote fp32 data updated under name: {}'.format(self.tag))
|
||||
|
||||
def _restore_fp32(self):
|
||||
""" Restore fp32 function config from remote database."""
|
||||
if self.type == 'client':
|
||||
time_left = self.time_limit
|
||||
while time_left:
|
||||
if self.fp32.count_documents({}):
|
||||
fp32_object = self.fp32.find({})
|
||||
_fp32_acc = object_loads(fp32_object[0]['fp32_acc'])
|
||||
_fp32_lat = object_loads(fp32_object[0]['fp32_lat'])
|
||||
logger.info('Remote fp32 data restored')
|
||||
return _fp32_acc, _fp32_lat
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote data (fp32): {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
time_left -= 1
|
||||
raise Exception('WARNING: Time limit for Remote reached!!! config name: {}'.format(
|
||||
self.tag))
|
||||
return None, None
|
||||
|
||||
def restore_remote_trials(self):
|
||||
""" Restore trials from remote database."""
|
||||
time_left = TRIALS_RESTORATION_TIME_LIMIT_S
|
||||
while time_left:
|
||||
if self.trials.count_documents({}):
|
||||
trials_object = self.trials.find({})
|
||||
trials_object_gfs = self.remote_fs.open_download_stream(trials_object[0]['file_id'])
|
||||
hpopt_trials = object_loads(trials_object_gfs.read())
|
||||
logger.info('Remote trials restored: {}'.format(len(hpopt_trials.trials)))
|
||||
return hpopt_trials
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote data (trials): {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
time_left -= 1
|
||||
raise Exception('WARNING: Time limit for Remote reached! config name: {}'.format(
|
||||
self.tag))
|
||||
|
||||
def _restore_remote_config(self):
|
||||
""" Restore loss function config from remote database."""
|
||||
time_left = self.time_limit
|
||||
while time_left:
|
||||
if self.params.count_documents({}):
|
||||
params_object = self.params.find({})
|
||||
config_valid = params_object[0]['valid']
|
||||
if config_valid:
|
||||
config = params_object[0]['config']
|
||||
logger.info('\nRemote params to be restored:\n\
|
||||
max_trials: {}\n\
|
||||
max_minutes: {}\n\
|
||||
accuracy_loss: {}\n\
|
||||
latency_reduce: {}\n\
|
||||
expected_quantization_ratio: {}\n\
|
||||
accuracy_weight: {}\n\
|
||||
latency_weight: {}\n\
|
||||
quantization_ratio_weight: {}\n\
|
||||
eval_subset_size: {}'.format(config.get('max_trials', None),
|
||||
config.get('max_minutes', None),
|
||||
config.get('accuracy_loss', 1),
|
||||
config.get('latency_reduce', 1),
|
||||
config.get('expected_quantization_ratio', 0.5),
|
||||
config.get('accuracy_weight', 1.0),
|
||||
config.get('latency_weight', 1.0),
|
||||
config.get('quantization_ratio_weight', 1.0),
|
||||
config.get('eval_subset_size', None)))
|
||||
return config
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Found old config. Waiting for config to be updated by server.')
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote config: {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
time_left -= 1
|
||||
raise Exception('WARNING: Time limit for Remote reached!!! config name: {}'.format(
|
||||
self.tag))
|
||||
|
||||
def restore_remote_search_space(self):
|
||||
""" Restore search_space from remote database."""
|
||||
time_left = self.time_limit
|
||||
while time_left:
|
||||
if self.search_space.count_documents({}):
|
||||
search_space_object = self.search_space.find({})
|
||||
search_space = object_loads(search_space_object[0]['data'])
|
||||
logger.info('Remote search_space restored')
|
||||
return search_space
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote data (search_space): {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
time_left -= 1
|
||||
raise Exception('WARNING: Time limit for Remote reached!!! config name: {}'.format(
|
||||
self.tag))
|
||||
|
||||
def _update_remote_config(self, _config=None, valid=True):
|
||||
""" Update remote loss function config."""
|
||||
if self.params.count_documents({}):
|
||||
self.params.update_one({}, {'$set': {'valid': valid}})
|
||||
else:
|
||||
self.params.insert_one({'config': _config, 'valid': valid})
|
||||
logger.info('Remote config updated under name: {}'.format(self.tag))
|
||||
|
||||
def update_remote_search_space(self, _search_space):
|
||||
""" Update remote search space."""
|
||||
self.search_space.insert_one({'data': object_dumps(_search_space)})
|
||||
logger.info('Remote search space updated under name: {}'.format(self.tag))
|
||||
|
||||
def update_remote_trials(self, _hpopt_trials):
|
||||
""" Upload local trials to remote database.
|
||||
- if some trials already in database, update only last trial and merge it,
|
||||
- if not, upload all local trials to remote database (for warm start mode),
|
||||
"""
|
||||
remote_data_updated = False
|
||||
retry_time_left = self.unlock_time_limit_s
|
||||
while not remote_data_updated and retry_time_left:
|
||||
retry_time_left -= 1
|
||||
# Update remote data if exist, if not create new remote config
|
||||
if self.trials.count_documents({'app': 'tpe'}):
|
||||
trials_object_remote_s = self.trials.find({'app': 'tpe'})
|
||||
if trials_object_remote_s[0]['Lock']:
|
||||
logger.info('Remote collection locked. Waiting for unlock')
|
||||
if not retry_time_left:
|
||||
raise Exception('WARNING: Retry limit for Trial remote write reached!!!')
|
||||
time.sleep(1)
|
||||
else:
|
||||
self.trials.update_one({'app': 'tpe'}, {'$set': {'Lock': 1}})
|
||||
# GridFS get remote trials file
|
||||
current_file = trials_object_remote_s[0]['file_id']
|
||||
if not ObjectId.is_valid(current_file):
|
||||
raise Exception('Remote file corrupted!')
|
||||
trials_object_remote_s_gfs = self.remote_fs.open_download_stream(current_file)
|
||||
trials_object_remote_gfs = object_loads(trials_object_remote_s_gfs.read())
|
||||
# Merge last local trial with remote data
|
||||
_hpopt_trials = trials_from_docs(list(trials_object_remote_gfs) + [list(_hpopt_trials)[-1]])
|
||||
# Upload to database
|
||||
# GridFS upload new trials file
|
||||
new_file = self.remote_fs.upload_from_stream(
|
||||
self.tag,
|
||||
object_dumps(_hpopt_trials), metadata={'trials_count': len(_hpopt_trials.trials)})
|
||||
# Update trials info with new trials file_id
|
||||
self.trials.update_one({'app': 'tpe'}, {'$set': {'Lock': 0, 'file_id': new_file}})
|
||||
# Remove old remote trials file
|
||||
self.remote_fs.delete(current_file)
|
||||
logger.info('Remote trials updated. Total: {} (tag: {})'.format(len(_hpopt_trials.trials),
|
||||
self.tag))
|
||||
remote_data_updated = True
|
||||
return _hpopt_trials
|
||||
else:
|
||||
# GridFS write trials file
|
||||
new_file = self.remote_fs.upload_from_stream(self.tag, object_dumps(_hpopt_trials),
|
||||
metadata={'trials_count': len(_hpopt_trials.trials)})
|
||||
# unlock trials to be available for other nodes with current trials id
|
||||
self.trials.insert_one({'app': 'tpe', 'Lock': 0, 'file_id': new_file})
|
||||
logger.info('No remote trials. First write for config {}'.format(self.tag))
|
||||
remote_data_updated = True
|
||||
return _hpopt_trials
|
||||
|
||||
def update_remote_evaluated_params(self, _evaluated_params):
|
||||
""" Upload local evaluated params to remote database.
|
||||
- if some params already in database, update only last params and merge it,
|
||||
- if not, upload all local params to remote database (for warm start mode),
|
||||
"""
|
||||
remote_data_updated = False
|
||||
retry_time_left = self.unlock_time_limit_s
|
||||
while not remote_data_updated and retry_time_left:
|
||||
retry_time_left -= 1
|
||||
# Update remote data if exist, if not create new remote config
|
||||
if self.evaluated_params.count_documents({'app': 'tpe'}):
|
||||
params_object_remote_s = self.evaluated_params.find({'app': 'tpe'})
|
||||
if params_object_remote_s[0]['Lock']:
|
||||
logger.info('Remote collection locked. Waiting for unlock')
|
||||
if not retry_time_left:
|
||||
raise Exception('WARNING: Retry limit for evaluated parameters remote write reached!!!')
|
||||
time.sleep(1)
|
||||
else:
|
||||
self.evaluated_params.update_one({'app': 'tpe'}, {'$set': {'Lock': 1}})
|
||||
remote_params = object_loads(params_object_remote_s[0]['data'])
|
||||
if not isinstance(remote_params, list):
|
||||
raise Exception('Received remote parameters object is not a list!!!')
|
||||
_evaluated_params = list(remote_params) + [list(_evaluated_params)[-1]]
|
||||
self.evaluated_params.update_one(
|
||||
{'app': 'tpe'}, {'$set': {'Lock': 0, 'data': object_dumps(_evaluated_params)}})
|
||||
remote_data_updated = True
|
||||
logger.info('Remote evaluated parameters set updated under name: {}'.format(self.tag))
|
||||
return _evaluated_params
|
||||
else:
|
||||
self.evaluated_params.insert_one({'app': 'tpe', 'Lock': 0, 'data': object_dumps(_evaluated_params)})
|
||||
remote_data_updated = True
|
||||
logger.info('Remote evaluated parameters set updated under name: {}'.format(self.tag))
|
||||
return _evaluated_params
|
||||
|
||||
def restore_remote_evaluated_params(self):
|
||||
""" Restore evaluated_params from remote database."""
|
||||
time_left = self.time_limit
|
||||
while time_left:
|
||||
if self.evaluated_params.count_documents({}):
|
||||
evaluated_params_object = self.evaluated_params.find({})
|
||||
evaluated_params = object_loads(evaluated_params_object[0]['data'])
|
||||
logger.info('Remote evaluated_params restored')
|
||||
return evaluated_params
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote data (evaluated_params): {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
time_left -= 1
|
||||
raise Exception('WARNING: Time limit for Remote reached!!! config name: {}'.format(
|
||||
self.tag))
|
||||
|
||||
def request_remote_benchmark(self, model, iteration):
|
||||
""" For Clients only.
|
||||
Upload request with params for server to create model and run benchmark.
|
||||
Wait for response.
|
||||
"""
|
||||
if self.type == 'client' and self.mode == 'master':
|
||||
self._set_client_status(active=True)
|
||||
model_file_id, weights_file_id = self.upload_model(model, iteration)
|
||||
remote_id = self.model.insert_one({
|
||||
'name': self.name,
|
||||
'iter': iteration,
|
||||
'lat': 0,
|
||||
'file_id': model_file_id,
|
||||
'bin_file_id': weights_file_id})
|
||||
logger.info('Model queued for remote evaluation id: {}'.format(remote_id.inserted_id))
|
||||
time_left = 0
|
||||
remote_lat = 0
|
||||
while not remote_lat and time_left < self.time_limit:
|
||||
remote_params_obj = self.model.find({'_id':remote_id.inserted_id})[0]
|
||||
remote_lat = remote_params_obj['lat']
|
||||
time_left += 1
|
||||
if not time_left % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for remote benchmark: {}s'.format(time_left))
|
||||
time.sleep(1)
|
||||
if remote_lat:
|
||||
logger.info('remote_lat_client: {}'.format(remote_lat))
|
||||
self.model.delete_one({'_id' : remote_id.inserted_id})
|
||||
self.remote_fs.delete(model_file_id)
|
||||
self.remote_fs.delete(weights_file_id)
|
||||
return remote_lat
|
||||
return None
|
||||
|
||||
def upload_model(self, model, trial_no):
|
||||
tmp_dir, path_to_model, path_to_weights = self._create_temp_dir()
|
||||
save_model(model, tmp_dir, 'tmp_model')
|
||||
with open(path_to_model, 'rb') as file:
|
||||
model_file_id = self.remote_fs.upload_from_stream(
|
||||
self.tag,
|
||||
file, metadata={'iter': trial_no, 'type': 'model'})
|
||||
with open(path_to_weights, 'rb') as file:
|
||||
weights_file_id = self.remote_fs.upload_from_stream(
|
||||
self.tag,
|
||||
file, metadata={'iter': trial_no, 'type': 'weights'})
|
||||
return model_file_id, weights_file_id
|
||||
|
||||
def calculate_remote_requests(self):
|
||||
""" For Server only.
|
||||
Check in loop for requests from clients to run benchmark.
|
||||
Finish when all clients clear activation flag.
|
||||
"""
|
||||
if self.type == 'server' and self.mode == 'master':
|
||||
waiting_time = 1
|
||||
while self._check_clients_status(not waiting_time % REMOTE_DATA_INFO_FREQ_S):
|
||||
waiting_time += 1
|
||||
model_count = self.model.count_documents({'lat' : {'$eq': 0}})
|
||||
if not waiting_time % REMOTE_DATA_INFO_FREQ_S:
|
||||
logger.info('Waiting for requests: {}'.format(waiting_time))
|
||||
if model_count:
|
||||
logger.info('Models in queue: {}'.format(model_count))
|
||||
if model_count:
|
||||
remote_params_obj = self.model.find({'lat' : {'$eq': 0}})[0]
|
||||
remote_lat = remote_params_obj['lat']
|
||||
remote_iter = remote_params_obj['iter']
|
||||
remote_name = remote_params_obj['name']
|
||||
remote_file_id = remote_params_obj['file_id']
|
||||
remote_file_id_bin = remote_params_obj['bin_file_id']
|
||||
logger.info('Starting for: {} iter: {}'.format(remote_name, remote_iter))
|
||||
if remote_lat == 0:
|
||||
lat = self._run_remote_benchmark(remote_file_id, remote_file_id_bin)
|
||||
logger.info('name: {} remote_lat_res: {} for iter: {}'.format(remote_name, lat, remote_iter))
|
||||
self.model.update_one({'_id': remote_params_obj['_id']}, {'$set': {'lat': lat}})
|
||||
time.sleep(1)
|
||||
|
||||
def _create_temp_dir(self):
|
||||
__MODEL_PATH__ = create_tmp_dir(gettempdir())
|
||||
model_name = 'tmp_model'
|
||||
path_to_model = __MODEL_PATH__.name + '/' + model_name + '.xml'
|
||||
path_to_weights = __MODEL_PATH__.name + '/' + model_name + '.bin'
|
||||
return __MODEL_PATH__.name, path_to_model, path_to_weights
|
||||
|
||||
def _run_remote_benchmark(self, file_id, remote_file_id_bin):
|
||||
__MODEL_PATH__ = create_tmp_dir(gettempdir())
|
||||
model_name = 'tmp_model'
|
||||
path_to_model = __MODEL_PATH__.name + '/' + model_name + '.xml'
|
||||
path_to_weights = __MODEL_PATH__.name + '/' + model_name + '.bin'
|
||||
new_file = self.remote_fs.open_download_stream(file_id)
|
||||
new_file_bin = self.remote_fs.open_download_stream(remote_file_id_bin)
|
||||
with open(path_to_model, 'wb') as file:
|
||||
file.write(new_file.read())
|
||||
with open(path_to_weights, 'wb') as file:
|
||||
file.write(new_file_bin.read())
|
||||
lat = benchmark_embedded(mf=path_to_model)
|
||||
return lat
|
||||
|
||||
def _check_clients_status(self, log=False):
|
||||
""" Check if are any clients in 'active' state."""
|
||||
clients_number = self.clients.count_documents({})
|
||||
clients_active = self.clients.count_documents({'active' : {'$eq': 1}})
|
||||
if log:
|
||||
logger.info('Total/Active clients:{}/{}'.format(clients_number, clients_active))
|
||||
if clients_active:
|
||||
self.wait_for_client = False
|
||||
return 1 if self.wait_for_client else clients_active
|
||||
|
||||
def _set_client_status(self, active=False):
|
||||
""" Set or clear 'active' flag for clients."""
|
||||
if self.type == 'client':
|
||||
if self.id is None:
|
||||
self.id = self.clients.insert_one({'active': 1 if active else 0})
|
||||
logger.info('Setting client status to True')
|
||||
else:
|
||||
self.clients.update_one({'_id': self.id.inserted_id}, {'$set': {'active': 1 if active else 0}})
|
||||
if not active:
|
||||
logger.info('Setting client status to False')
|
||||
|
||||
def cleanup(self):
|
||||
""" Ending cleanup """
|
||||
if self.type == 'client':
|
||||
self._set_client_status(active=False)
|
||||
if self.type == 'server':
|
||||
self._update_remote_config(valid=False)
|
||||
@@ -1,55 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import builtins
|
||||
import collections
|
||||
import datetime
|
||||
import pickle
|
||||
|
||||
import io
|
||||
import numpy
|
||||
import numpy.core.multiarray
|
||||
import hyperopt.base
|
||||
import hyperopt.pyll.base
|
||||
|
||||
|
||||
def object_dump(obj, filename):
|
||||
with open(filename, 'wb') as f:
|
||||
pickle.dump(obj, f)
|
||||
|
||||
def object_dumps(obj):
|
||||
return pickle.dumps(obj)
|
||||
|
||||
def object_load(filename):
|
||||
with open(filename, 'rb') as f:
|
||||
return _SafeUnpickler(f).load()
|
||||
|
||||
def object_loads(obj):
|
||||
f = io.BytesIO(obj)
|
||||
return _SafeUnpickler(f).load()
|
||||
|
||||
class _SafeUnpickler(pickle.Unpickler):
|
||||
"""Safe unpickler forbidding globals functions and classess
|
||||
"""
|
||||
|
||||
def find_class(self, module, name):
|
||||
secure_class = None
|
||||
# Only allow secure classes
|
||||
if module == 'builtins' and name == 'set':
|
||||
secure_class = getattr(builtins, name)
|
||||
if module == 'collections':
|
||||
secure_class = getattr(collections, name)
|
||||
if module == 'datetime':
|
||||
secure_class = getattr(datetime, name)
|
||||
if module == 'numpy' and name in ['dtype', 'ndarray']:
|
||||
secure_class = getattr(numpy, name)
|
||||
if module == 'numpy.core.multiarray' and name in ['scalar', '_reconstruct']:
|
||||
secure_class = getattr(numpy.core.multiarray, name)
|
||||
if module == 'hyperopt.base' and name == 'Trials':
|
||||
secure_class = getattr(hyperopt.base, name)
|
||||
if module == 'hyperopt.pyll.base' and name in ['Apply', 'Literal']:
|
||||
secure_class = getattr(hyperopt.pyll.base, name)
|
||||
if secure_class:
|
||||
return secure_class
|
||||
raise pickle.UnpicklingError(
|
||||
'global "%s.%s" is forbidden' % (module, name))
|
||||
@@ -1,33 +0,0 @@
|
||||
# Copyright (C) 2020-2021 Intel Corporation
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
from openvino.tools.pot.benchmark.benchmark import benchmark_embedded, set_benchmark_config
|
||||
from openvino.tools.pot.utils.logger import get_logger, init_logger
|
||||
from .utils.path import TEST_ROOT
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
REFERENCE_MODELS_PATH = TEST_ROOT/'../thirdparty/open_model_zoo/tools/accuracy_checker/data/test_models/SampLeNet.xml'
|
||||
|
||||
def test_benchmark(model=None, cfg=None):
|
||||
init_logger(level='INFO')
|
||||
if cfg:
|
||||
set_benchmark_config(cfg)
|
||||
if model:
|
||||
benchmark_embedded(model=model)
|
||||
return
|
||||
|
||||
path_to_model_file = str(REFERENCE_MODELS_PATH)
|
||||
logger.info('Benchmark test with {}'.format(path_to_model_file))
|
||||
|
||||
cfg = {'nireq': 0}
|
||||
set_benchmark_config(cfg)
|
||||
benchmark_embedded(model=None, mf=path_to_model_file, duration_seconds=1)
|
||||
|
||||
cfg = {'nireq': 0, 'benchmark_app_dir':""}
|
||||
set_benchmark_config(cfg)
|
||||
benchmark_embedded(model=None, mf=path_to_model_file, duration_seconds=1)
|
||||
|
||||
cfg = {'nireq': 0, 'benchmark_app_dir':"wrong_benchmark_dir"}
|
||||
set_benchmark_config(cfg)
|
||||
benchmark_embedded(model=None, mf=path_to_model_file, duration_seconds=1)
|
||||
Reference in New Issue
Block a user