openvino/docs/ops/image/Interpolate_4.md

Interpolate

Versioned name: Interpolate-4

Category: Image processing

Short description: Interpolate layer performs interpolation of independent slices in input tensor by specified dimensions and attributes.

Attributes

• mode

  • Description: specifies type of interpolation
  • Range of values: one of nearest, linear, linear_onnx, cubic
  • Type: string
  • Default value: none
  • Required: yes

• coordinate_transformation_mode

  • Description: specifies how to transform the coordinate in the resized tensor to the coordinate in the original tensor
  • Range of values: name of the transformation mode in string format (here scale[x] is output_shape[x] / input_shape[x] and x_resized is a coordinate in axis x, for any axis x from the input axes):
    • half_pixel - the coordinate in the original tensor axis x is calculated as ((x_resized + 0.5) / scale[x]) - 0.5.
    • pytorch_half_pixel - the coordinate in the original tensor axis x is calculated by (x_resized + 0.5) / scale[x] - 0.5 if output_shape[x] > 1 else 0.0.
    • asymmetric - the coordinate in the original tensor axis x is calculated according to the formula x_resized / scale[x].
    • tf_half_pixel_for_nn - the coordinate in the original tensor axis x is (x_resized + 0.5) / scale[x].
    • align_corners - the coordinate in the original tensor axis x is calculated as 0 if output_shape[x] == 1 else x_resized * (input_shape[x] - 1) / (output_shape[x] - 1).
  • Type: string
  • Default value: half_pixel
  • Required: no

• nearest_mode

  • Description: specifies round mode when mode == nearest and is used only when mode == nearest.
  • Range of values: name of the round mode in string format:
    • round_prefer_floor - this mode is known as round half down.
    • round_prefer_ceil - it is round half up mode.
    • floor - this mode computes the largest integer value not greater than rounded value.
    • ceil - this mode computes the smallest integer value not less than rounded value.
    • simple - this mode behaves as ceil mode when Interpolate is downsample, and as dropping the fractional part otherwise.
  • Type: string
  • Default value: round_prefer_floor
  • Required: no

• antialias

  • Description: antialias is a flag that specifies whether to perform anti-aliasing.
  • Range of values:
    • False - do not perform anti-aliasing
    • True - perform anti-aliasing
  • Type: boolean
  • Default value: False
  • Required: no

• pads_begin

  • Description: pads_begin specifies the number of pixels to add to the beginning of the image being interpolated. This addition of pixels is done before interpolation calculation.
  • Range of values: list of non-negative integer numbers
  • Type: int[]
  • Default value: [0]
  • Required: no

• pads_end

  • Description: pads_end specifies the number of pixels to add to the end of the image being interpolated. This addition of pixels is done before interpolation calculation.
  • Range of values: list of non-negative integer numbers
  • Type: int[]
  • Default value: [0]
  • Required: no

• cube_coeff

• Description: cube_coeff specifies the parameter a for cubic interpolation (see, e.g. article). cube_coeff is used only when mode == cubic.

  • Range of values: floating point number
  • Type: any of supported floating point type
  • Default value: -0.75
  • Required: no

Inputs

• 1: data - Input tensor with data for interpolation. Type of elements is any supported floating point type or int8 type. Required.

• 2: target_spatial_shape - 1D tensor describing output shape for spatial axes. Number of elements matches the number of indices in axes input, the order matches as well. Required.

• 3: axes - 1D tensor specifying dimension indices where interpolation is applied, and axes is any unordered list of indices of different dimensions of input tensor, e.g. [0, 4], [4, 0], [4, 2, 1], [1, 2, 3]. These indices should be non-negative integers from 0 to rank(data) - 1 inclusively. Other dimensions do not change. The order of elements in axes attribute matters, and mapped directly to elements in the 2nd input target_spatial_shape. Namely, output_shape[axes[i]] = target_spatial_shape[i] for all i in range(0, len(axes)) and output_shape[j] = input_shape[j] + pads_begin[j] + pads_end[j] for j not in axes, j in range(0, rank(data)). Optional with default value [0,...,rank(data) - 1].

Outputs

• 1: Resulting interpolated tensor with elements of the same type as input data tensor. The shape of the output matches input data shape except spatial dimensions mentioned in axes attribute. For other dimensions shape matches sizes from target_spatial_shape in order specified in axes.

Detailed description Calculations are performed according to the following rules.

import math
import numpy as np

class GetNearestPixel:
    def __init__(self, mode: str):
        self.func = {
            'round_prefer_floor': GetNearestPixel.prefer_floor_func,
            'round_prefer_ceil': GetNearestPixel.prefer_ceil_func,
            'floor': GetNearestPixel.floor_func,
            'ceil': GetNearestPixel.ceil_func,
            'simple': GetNearestPixel.simple_func
        }[mode]

    def __call__(self, x_original, is_downsample):
        return self.func(x_original, is_downsample)

    @staticmethod
    def prefer_floor_func(x_original, is_downsample):
        if x_original == int(x_original) + 0.5:
            return int(math.floor(x_original))
        else:
            return int(round(x_original))

    @staticmethod
    def prefer_ceil_func(x_original, is_downsample):
        return int(round(x_original))

    @staticmethod
    def floor_func(x_original, is_downsample):
        return int(math.floor(x_original))

    @staticmethod
    def ceil_func(x_original, is_downsample):
        return int(math.ceil(x_original))

    @staticmethod
    def simple_func(x_original, is_downsample):
        if is_downsample:
            return int(math.ceil(x_original))
        else:
            return int(x_original)


class GetOriginalCoordinate:
    def __init__(self, mode: str):
        self.func = {
            'half_pixel': GetOriginalCoordinate.half_pixel_func,
            'pytorch_half_pixel': GetOriginalCoordinate.pytorch_half_pixel_func,
            'asymmetric': GetOriginalCoordinate.asymmetric_func,
            'tf_half_pixel_for_nn': GetOriginalCoordinate.tf_half_pixel_for_nn_func,
            'align_corners': GetOriginalCoordinate.align_corners_func
        }[mode]

    def __call__(self, resized, x_scale, length_resized, length_original):
        return self.func(resized, x_scale, length_resized, length_original)

    @staticmethod
    def half_pixel_func(resized, x_scale, length_resized, length_original):
        return ((x_resized + 0.5) / x_scale) - 0.5

    @staticmethod
    def pytorch_half_pixel_func(resized, x_scale, length_resized, length_original):
        return (x_resized + 0.5) / x_scale - 0.5 if  length_resized > 1 else 0.0

    @staticmethod
    def asymmetric_func(resized, x_scale, length_resized, length_original):
        return x_resized / x_scale

    @staticmethod
    def tf_half_pixel_for_nn_func(resized, x_scale, length_resized, length_original):
        return (x_resized + 0.5) / x_scale

    @staticmethod
    def align_corners_func(resized, x_scale, length_resized, length_original):
        return  0 if length_resized == 1 else  x_resized * (length_original - 1) / (length_resized - 1)


def get_cubic_coeff(s, a):
    abs_s = abs(s)
    coeff = np.zeros(4)
    coeff[0] = a * (abs_s - 1.0) * (abs_s - 1.0) * abs_s
    coeff[1] = ((a + 2.0) * abs_s - (a + 3.0)) * abs_s * abs_s + 1.0
    coeff[2] = (((-a -2.0) * abs_s+ (2.0 * a + 3.0)) * abs_s - a) * abs_s
    coeff[3] = - a * abs_s * abs_s * (abs_s - 1.0)
    return coeff


def  triangle_coeffs(dz):
    return np.maximum(0.0, 1.0 - np.abs(dz))


class InterpolateCalculation:
    def __init__(self, attrs: dict):
        self.func = {
            'nearest': self.nearest_interpolation,
            'linear': self.linear_interpolation,
            'cubic': self.cubic_interpolation,
            'linear_onnx': self.onnx_linear_interpolation
        }['mode']

        if not('pads_begin' in attrs):
            self.pads_begin = [0]
        else:
            self.pads_begin = attrs['pads_begin']

        if not('pads_end' in attrs):
            self.pads_end = [0]
        else:
            self.pads_end = attrs['pads_end']

        if not ('coordinate_transformation_mode' in attrs):
            self.coordinate_transformation_mode = 'half_pixel'
        else:
            self.coordinate_transformation_mode = attrs['coordinate_transformation_mode']

        if ('align_corners' in attrs) and attrs['align_corners']:
            self.coordinate_transformation_mode = 'align_corners'

        if not ('nearest_mode' in attrs):
            self.nearest_mode = 'round_prefer_floor'
        else:
            self.nearest_mode = attrs['nearest_mode']

        if not ('cube_coeff' in attrs):
            self.cube_coeff = -0.75
        else:
            self.cube_coeff = attrs['cube_coeff']

        if not ('antialias' in self.attrs):
            self.antialias = False
        else:
            self.antialias = attrs['antialias']

        self.get_original_coordinate = self.get_coordinate_transformation_mode()


    def get_coordinate_transformation_mode(self):
        return GetOriginalCoordinate(self.coordinate_transformation_mode)

    def shape_infer(self, input_data, target_spatial_shape):
        result = input_data.shape + self.pads_begin + self.pads_end
        for i in range(0, len(self.axes)):
            result[self.axes[i]] = target_spatial_shape[i]
        return result

    @staticmethod
    def correct_pad(pad, rank):
        pad_len = len(pad)
        if pad_len < rank:
            return np.pad(pad, (0, rank - pad_len)).astype(np.int64)
        elif pad_len > rank:
            return np.array(pad[: rank - 1]).astype(np.int64)
        else:
            return np.array(pad, dtype=np.int64)

    def __call__(self, input_data, target_spatial_shape, axes):
        rank = input_data.ndim
        self.pads_begin = InterpolateCalculation.correct_pad(self.pads_begin, rank)
        self.pads_end = InterpolateCalculation.correct_pad(self.pads_end, rank)
        self.pads = list(zip(self.pads_begin, self.pads_end))
        self.axes = np.array(axes).astype(np.int64)

        self.output_shape = self.shape_infer(input_data, target_spatial_shape)
        padded_data = np.pad(input_data, self.pads)
        self.scales = self.output_shape / padded_data.shape
        self.input_shape = padded_data.shape
        return self.func(padded_data)

    def clip_coord(self, coord, axis):
        return max(0, min(coord, self.input_shape[axis] - 1))

    def cubic_interpolation(self, input_data):
        result = np.zeros(self.output_shape)
        num_of_axes = len(self.axes)
        indices = np.ndindex(tuple(4 for _ in range(num_of_axes)))
        for coordinates in np.ndindex(self.output_shape):
            for index in indices:
                input_coords = np.array(coordinates, dtype=np.int64)
                cubic_coeffs = []
                for i in range(len(index)):
                    axis = self.axes[i]
                    in_coord = self.get_original_coordinate(coordinates[axis], self.scales[axis], self.output_shape[axis], self.input_shape[axis])
                    cubic_coeffs.append(get_cubic_coeff(in_coord - math.floor(in_coord), self.cube_coeff))
                    input_coords[axis] = self.clip_coord(input_coords[axis] + index[i] - 1)
                data = input_data[input_coords]
                for i in range(len(index)):
                    data = data * cubic_coeffs[i][index[i]]
                result[coordinates] += data
        return result

    def linear_interpolation(self, input_data):
        result = np.zeros(self.output_shape)
        num_of_axes = len(self.axes)
        is_downsample = False

        for i in range(num_of_axes):
            is_downsample = is_downsample or (self.scales[self.axes[i]] < 1)

        antialias = is_downsample and self.antialias

        a = np.zeros(num_of_axes)
        for i in range(num_of_axes):
            a[i] = self.scales[self.axes[i]] if antialias else 1.0

        prod_of_a = np.prod(a)
        r = np.zeros(num_of_axes).astype(np.int64)
        for i in range(num_of_axes):
            r[i] = 2 if self.scales[self.axes[i]] > 1.0 else int(math.ceil(2.0/a[i]))

        indices = np.ndindex(2 * r + 1)

        for coordinates in np.ndindex(self.output_shape):
            sum = 0
            wsum = 0

            icoords = np.array(coordinates).astype(np.float64)
            for i in range(num_of_axes):
                axis = self.axes[i]
                in_coord = self.get_original_coordinate(coordinates[axis],  self.scales[axis], self.output_shape[axis], self.input_shape[axis])
                icoords[axis] = in_coord
            icoords_r = np.around(icoords).astype(np.int64)

            for index in indices:
                iarray = np.array(index).astype(np.int64) - r + input_coords[self.axes]
                conditions = [iarray[i] >= 0 and iarray[i] < self.input_shape[self.axes[i]] for i in range(num_of_axes)]
                if not all(conditions):
                    continue

                dz = icoords[self.axes] - iarray
                w = prod_of_a * np.prod(triangle_coeffs(dz))
                wsum += w
                input_indices = np.array(coordinates).astype
                input_indices[self.axes] = iarray
                sum += w * input_data[input_indices]

            result[coordinates] = sum / wsum

        return result

    def onnx_linear_interpolation(self, input_data):
        rank = len(self.input_shape)
        assert rank in [2, 4], "mode 'linear_onnx' supports only 2D or 4D tensors"
        assert set(self.axes) == {2, 3} or set(self.axes) == {0, 1}, \
            "mode 'linear_onnx' supports only case when axes = {2, 3} or axes = {0, 1}"

        result = np.zeros(self.output_shape)

        if rank == 2:
            reshaped_data = np.reshape(input_data, (1, 1, self.input_shape[0], self.input_shape[1]))
            result = np.reshape(result,  (1, 1, self.output_shape[0], self.output_shape[1]))
        else:
            reshaped_data = input_data

        output_height = self.output_shape[0] if rank == 2 else self.output_shape[2]
        output_width = self.output_shape[1] if rank == 2 else self.output_shape[3]
        input_height = self.input_shape[0] if rank == 2 else self.input_shape[2]
        input_width = self.input_shape[1] if rank == 2 else self.input_shape[3]
        height_scale = self.scales[0] if rank == 2 else self.scales[2]
        width_scale = self.scales[1] if rank == 2 else self.scales[3]
        batch_size = 1 if rank == 2 else self.input_shape[0]
        num_channels = 1 if rank == 2 else self.input_shape[1]

        in_y1 = np.zeros(output_height).astype(np.int64)
        in_y2 = np.zeros(output_height).astype(np.int64)
        in_x1 = np.zeros(output_width).astype(np.int64)
        in_x2 = np.zeros(output_width).astype(np.int64)

        dy1 = np.zeros(output_height).astype(np.float64)
        dy2 = np.zeros(output_height).astype(np.float64)
        dx1 = np.zeros(output_width).astype(np.float64)
        dx2 = np.zeros(output_width).astype(np.float64)

        y_original = np.zeros(output_height).astype(np.float64)
        x_original = np.zeros(output_width).astype(np.float64)

        for y in range(output_height):
            in_y = self.get_original_coordinate(y, height_scale, output_height, input_height)
            y_original[y] = in_y
            in_y = max(0, min(in_y, input_height - 1))
            in_y1[y] = max(0, min(int(in_y), input_height - 1))
            in_y2[y] = min(in_y1[y] + 1, input_height - 1)
            dy1[y] = abs(in_y - in_y1[y])
            dy2[y] = abs(in_y - in_y2[y])

            if in_y1 == in_y2:
                dy1[y] = 0.5
                dy2[y] = 0.5

        for x in range(output_width):
            in_x = self.get_original_coordinate(x, width_scale, output_width, input_width)
            x_original[x] = in_x
            in_x = max(0, min(in_x, input_width - 1))
            in_x1[x] = max(0, min(int(in_x), input_width - 1))
            in_x2[x] = min(in_x1[x] + 1, input_width - 1)
            dx1[x] = abs(in_x - in_x1[x])
            dx2[x] = abs(in_x - in_x2[x])

            if in_x1 == in_x2:
                dx1[x] = 0.5
                dx2[x] = 0.5

        for n in range(batch_size):
            for c in range(num_channels):
                for y in range(output_height):
                    for x in range(output_width):
                        x11 = reshaped_data[n, c, in_y1[y], in_x1[x]]
                        x21 = reshaped_data[n, c, in_y1[y], in_x2[x]]
                        x12 = reshaped_data[n, c, in_y2[y], in_x1[x]]
                        x22 = reshaped_data[n, c, in_y2[y], in_x2[x]]
                        temp = dx2[x] * dy2[y] * x11 + dx1[x] * dy2[y] * x21
                        temp += dx2[x] * dy1[y] * x12 + dx1[x] * dy1[y] * x22
                        result[n, c, y, x] = temp

        return np.reshape(result, self.output_shape)

    def nearest_interpolation(self, input_data):
        if not ('nearest_mode' in self.attrs):
            self.attrs['nearest_mode'] = 'floor'

        self.get_nearest_pixel = GetNearestPixel(attrs['nearest_mode'])

        result = np.zeros(self.output_shape)

        num_of_axes = len(self.axes)
        for coordinates in np.ndindex(self.output_shape):
            input_coords = np.array(coordinates, dtype=np.int64)
            for i in range(num_of_axes):
                axis = self.axes[i]
                in_coord = self.get_original_coordinate(coordinates[axis], self.scales[axis], self.output_shape[axis], self.input_shape[axis])
                nearest_pixel = self.get_nearest_pixel(in_coord, self.scales[axis] < 1)
                input_coords[axis] = max(0, min(nearest_pixel, self.input_shape[axis] - 1))
           result[coordinates] = input_data[input_coords]

        return result

Example

<layer ... type="Interpolate" ...>
    <data axes="2,3" align_corners="0" pads_begin="0" pads_end="0" mode="linear"/>
    <input>
        <port id="0">
            <dim>1</dim>
            <dim>2</dim>
            <dim>48</dim>
            <dim>80</dim>
        </port>
        <port id="1">
            <dim>2</dim>  <!--The values in this input are [50, 60] -->
        </port>
    </input>
    <output>
        <port id="0">
            <dim>1</dim>
            <dim>2</dim>
            <dim>50</dim>
            <dim>60</dim>
        </port>
    </output>
</layer>