AUGRU op ov core shell and reference implementation (#12163)
This commit is contained in:
Katarzyna Mitrus authored and 
GitHub committed
1 parent
371e393581
commit
67f4b8ee66
19 files changed
+2267
-23
No files matched your search
@@ -0,0 +1,135 @@
|
||||
# AUGRUCell
|
||||
|
||||
**Versioned name**: *AUAUGRUCell*
|
||||
|
||||
**Category**: *Sequence processing*
|
||||
|
||||
**Short description**: *AUGRUCell* represents a single AUGRU Cell (GRU with attentional update gate).
|
||||
|
||||
**Detailed description**: The main difference between *AUGRUCell* and [GRUCell](../../../../../docs/ops/sequence/GRUCell_3.md) is the additional attention score input `A`, which is a multiplier for the update gate.
|
||||
The AUGRU formula is based on the [paper arXiv:1809.03672](https://arxiv.org/abs/1809.03672).
|
||||
|
||||
```
|
||||
AUGRU formula:
|
||||
* - matrix multiplication
|
||||
(.) - Hadamard product (element-wise)
|
||||
|
||||
f, g - activation functions
|
||||
z - update gate, r - reset gate, h - hidden gate
|
||||
a - attention score
|
||||
|
||||
rt = f(Xt*(Wr^T) + Ht-1*(Rr^T) + Wbr + Rbr)
|
||||
zt = f(Xt*(Wz^T) + Ht-1*(Rz^T) + Wbz + Rbz)
|
||||
ht = g(Xt*(Wh^T) + (rt (.) Ht-1)*(Rh^T) + Rbh + Wbh) # 'linear_before_reset' is False
|
||||
|
||||
zt' = (1 - at) (.) zt # multiplication by attention score
|
||||
|
||||
Ht = (1 - zt') (.) ht + zt' (.) Ht-1
|
||||
```
|
||||
|
||||
**Attributes**
|
||||
|
||||
* *hidden_size*
|
||||
|
||||
* **Description**: *hidden_size* specifies hidden state size.
|
||||
* **Range of values**: a positive integer
|
||||
* **Type**: `int`
|
||||
* **Required**: *yes*
|
||||
|
||||
* *activations*
|
||||
|
||||
* **Description**: activation functions for gates
|
||||
* **Range of values**: *sigmoid*, *tanh*
|
||||
* **Type**: a list of strings
|
||||
* **Default value**: *sigmoid* for f, *tanh* for g
|
||||
* **Required**: *no*
|
||||
|
||||
* *activations_alpha, activations_beta*
|
||||
|
||||
* **Description**: *activations_alpha, activations_beta* attributes of functions; applicability and meaning of these attributes depends on chosen activation functions
|
||||
* **Range of values**: []
|
||||
* **Type**: `float[]`
|
||||
* **Default value**: []
|
||||
* **Required**: *no*
|
||||
|
||||
* *clip*
|
||||
|
||||
* **Description**: *clip* specifies bound values *[-C, C]* for tensor clipping. Clipping is performed before activations.
|
||||
* **Range of values**: `0.`
|
||||
* **Type**: `float`
|
||||
* **Default value**: `0.` that means the clipping is not applied
|
||||
* **Required**: *no*
|
||||
|
||||
* *linear_before_reset*
|
||||
|
||||
* **Description**: *linear_before_reset* flag denotes, if the output of hidden gate is multiplied by the reset gate before or after linear transformation.
|
||||
* **Range of values**: False
|
||||
* **Type**: `boolean`
|
||||
* **Default value**: False
|
||||
* **Required**: *no*.
|
||||
|
||||
**Inputs**
|
||||
|
||||
* **1**: `X` - 2D tensor of type *T* and shape `[batch_size, input_size]`, input data. **Required.**
|
||||
|
||||
* **2**: `H_t` - 2D tensor of type *T* and shape `[batch_size, hidden_size]`. Input with initial hidden state data. **Required.**
|
||||
|
||||
* **3**: `W` - 2D tensor of type *T* and shape `[3 * hidden_size, input_size]`. The weights for matrix multiplication, gate order: zrh. **Required.**
|
||||
|
||||
* **4**: `R` - 2D tensor of type *T* and shape `[3 * hidden_size, hidden_size]`. The recurrence weights for matrix multiplication, gate order: zrh. **Required.**
|
||||
|
||||
* **6**: `B` - 2D tensor of type *T*. The biases. If *linear_before_reset* is set to `False`, then the shape is `[3 * hidden_size]`, gate order: zrh. Otherwise the shape is `[4 * hidden_size]` - the sum of biases for z and r gates (weights and recurrence weights), the biases for h gate are placed separately. **Required.**
|
||||
|
||||
* **7**: `A` - 2D tensor of type *T* and shape `[batch_size, 1]`, the attention score. **Required.**
|
||||
|
||||
|
||||
**Outputs**
|
||||
|
||||
* **1**: `Ho` - 2D tensor of type *T* `[batch_size, hidden_size]`, the last output value of hidden state.
|
||||
|
||||
**Types**
|
||||
|
||||
* *T*: any supported floating-point type.
|
||||
|
||||
**Example**
|
||||
```xml
|
||||
<layer ... type="AUGRUCell" ...>
|
||||
<data hidden_size="128"/>
|
||||
<input>
|
||||
<port id="0"> <!-- `X` input data -->
|
||||
<dim>1</dim>
|
||||
<dim>16</dim>
|
||||
</port>
|
||||
<port id="1"> <!-- `H_t` input -->
|
||||
<dim>1</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="3"> <!-- `W` weights input -->
|
||||
<dim>384</dim>
|
||||
<dim>16</dim>
|
||||
</port>
|
||||
<port id="4"> <!-- `R` recurrence weights input -->
|
||||
<dim>384</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="5"> <!-- `B` bias input -->
|
||||
<dim>384</dim>
|
||||
</port>
|
||||
<port id="6"> <!-- `A` attention score input -->
|
||||
<dim>1</dim>
|
||||
<dim>1</dim>
|
||||
</port>
|
||||
</input>
|
||||
<output>
|
||||
<port id="7"> <!-- `Y` output -->
|
||||
<dim>1</dim>
|
||||
<dim>4</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="8"> <!-- `Ho` output -->
|
||||
<dim>1</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
</output>
|
||||
</layer>
|
||||
```
|
||||
@@ -0,0 +1,161 @@
|
||||
# AUGRUSequence
|
||||
|
||||
**Versioned name**: *AUGRUSequence*
|
||||
|
||||
**Category**: *Sequence processing*
|
||||
|
||||
**Short description**: *AUGRUSequence* operation represents a series of AUGRU cells (GRU with attentional update gate).
|
||||
|
||||
**Detailed description**: The main difference between *AUGRUSequence* and [GRUSequence](../../../../../docs/ops/sequence/GRUSequence_5.md) is the additional attention score input `A`, which is a multiplier for the update gate.
|
||||
The AUGRU formula is based on the [paper arXiv:1809.03672](https://arxiv.org/abs/1809.03672).
|
||||
|
||||
```
|
||||
AUGRU formula:
|
||||
* - matrix multiplication
|
||||
(.) - Hadamard product (element-wise)
|
||||
|
||||
f, g - activation functions
|
||||
z - update gate, r - reset gate, h - hidden gate
|
||||
a - attention score
|
||||
|
||||
rt = f(Xt*(Wr^T) + Ht-1*(Rr^T) + Wbr + Rbr)
|
||||
zt = f(Xt*(Wz^T) + Ht-1*(Rz^T) + Wbz + Rbz)
|
||||
ht = g(Xt*(Wh^T) + (rt (.) Ht-1)*(Rh^T) + Rbh + Wbh) # 'linear_before_reset' is False
|
||||
|
||||
zt' = (1 - at) (.) zt # multiplication by attention score
|
||||
|
||||
Ht = (1 - zt') (.) ht + zt' (.) Ht-1
|
||||
```
|
||||
|
||||
Activation functions for gates: *sigmoid* for f, *tanh* for g.
|
||||
Only `forward` direction is supported, so `num_directions` is always equal to `1`.
|
||||
|
||||
**Attributes**
|
||||
|
||||
* *hidden_size*
|
||||
|
||||
* **Description**: *hidden_size* specifies hidden state size.
|
||||
* **Range of values**: a positive integer
|
||||
* **Type**: `int`
|
||||
* **Required**: *yes*
|
||||
|
||||
* *activations*
|
||||
|
||||
* **Description**: activation functions for gates
|
||||
* **Range of values**: *sigmoid*, *tanh*
|
||||
* **Type**: a list of strings
|
||||
* **Default value**: *sigmoid* for f, *tanh* for g
|
||||
* **Required**: *no*
|
||||
|
||||
* *activations_alpha, activations_beta*
|
||||
|
||||
* **Description**: *activations_alpha, activations_beta* attributes of functions; applicability and meaning of these attributes depends on chosen activation functions
|
||||
* **Range of values**: []
|
||||
* **Type**: `float[]`
|
||||
* **Default value**: []
|
||||
* **Required**: *no*
|
||||
|
||||
* *clip*
|
||||
|
||||
* **Description**: *clip* specifies bound values *[-C, C]* for tensor clipping. Clipping is performed before activations.
|
||||
* **Range of values**: `0.`
|
||||
* **Type**: `float`
|
||||
* **Default value**: `0.` that means the clipping is not applied
|
||||
* **Required**: *no*
|
||||
|
||||
* *direction*
|
||||
|
||||
* **Description**: Specify if the RNN is forward, reverse, or bidirectional. If it is one of *forward* or *reverse* then `num_directions = 1`, if it is *bidirectional*, then `num_directions = 2`. This `num_directions` value specifies input/output shape requirements.
|
||||
* **Range of values**: *forward*
|
||||
* **Type**: `string`
|
||||
* **Default value**: *forward*
|
||||
* **Required**: *no*
|
||||
|
||||
* *linear_before_reset*
|
||||
|
||||
* **Description**: *linear_before_reset* flag denotes, if the output of hidden gate is multiplied by the reset gate before or after linear transformation.
|
||||
* **Range of values**: False
|
||||
* **Type**: `boolean`
|
||||
* **Default value**: False
|
||||
* **Required**: *no*
|
||||
|
||||
**Inputs**
|
||||
|
||||
* **1**: `X` - 3D tensor of type *T1* `[batch_size, seq_length, input_size]`, input data. **Required.**
|
||||
|
||||
* **2**: `H_t` - 3D tensor of type *T1* and shape `[batch_size, num_directions, hidden_size]`. Input with initial hidden state data. **Required.**
|
||||
|
||||
* **3**: `sequence_lengths` - 1D tensor of type *T2* and shape `[batch_size]`. Specifies real sequence lengths for each batch element. **Required.**
|
||||
|
||||
* **4**: `W` - 3D tensor of type *T1* and shape `[num_directions, 3 * hidden_size, input_size]`. The weights for matrix multiplication, gate order: zrh. **Required.**
|
||||
|
||||
* **5**: `R` - 3D tensor of type *T1* and shape `[num_directions, 3 * hidden_size, hidden_size]`. The recurrence weights for matrix multiplication, gate order: zrh. **Required.**
|
||||
|
||||
* **6**: `B` - 2D tensor of type *T1*. The biases. If *linear_before_reset* is set to `False`, then the shape is `[num_directions, 3 * hidden_size]`, gate order: zrh. Otherwise the shape is `[num_directions, 4 * hidden_size]` - the sum of biases for z and r gates (weights and recurrence weights), the biases for h gate are placed separately. **Required.**
|
||||
|
||||
* **7**: `A` - 3D tensor of type *T1* `[batch_size, seq_length, 1]`, the attention score. **Required.**
|
||||
|
||||
**Outputs**
|
||||
|
||||
* **1**: `Y` - 4D tensor of type *T1* `[batch_size, num_directions, seq_length, hidden_size]`, concatenation of all the intermediate output values of the hidden.
|
||||
|
||||
* **2**: `Ho` - 3D tensor of type *T1* `[batch_size, num_directions, hidden_size]`, the last output value of hidden state.
|
||||
|
||||
**Types**
|
||||
|
||||
* *T1*: any supported floating-point type.
|
||||
* *T2*: any supported integer type.
|
||||
|
||||
**Example**
|
||||
```xml
|
||||
<layer ... type="AUGRUSequence" ...>
|
||||
<data hidden_size="128"/>
|
||||
<input>
|
||||
<port id="0"> <!-- `X` input data -->
|
||||
<dim>1</dim>
|
||||
<dim>4</dim>
|
||||
<dim>16</dim>
|
||||
</port>
|
||||
<port id="1"> <!-- `H_t` input -->
|
||||
<dim>1</dim>
|
||||
<dim>1</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="2"> <!-- `sequence_lengths` input -->
|
||||
<dim>1</dim>
|
||||
</port>
|
||||
<port id="3"> <!-- `W` weights input -->
|
||||
<dim>1</dim>
|
||||
<dim>384</dim>
|
||||
<dim>16</dim>
|
||||
</port>
|
||||
<port id="4"> <!-- `R` recurrence weights input -->
|
||||
<dim>1</dim>
|
||||
<dim>384</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="5"> <!-- `B` bias input -->
|
||||
<dim>1</dim>
|
||||
<dim>384</dim>
|
||||
</port>
|
||||
<port id="6"> <!-- `A` attention score input -->
|
||||
<dim>1</dim>
|
||||
<dim>4</dim>
|
||||
<dim>1</dim>
|
||||
</port>
|
||||
</input>
|
||||
<output>
|
||||
<port id="7"> <!-- `Y` output -->
|
||||
<dim>1</dim>
|
||||
<dim>1</dim>
|
||||
<dim>4</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
<port id="8"> <!-- `Ho` output -->
|
||||
<dim>1</dim>
|
||||
<dim>1</dim>
|
||||
<dim>128</dim>
|
||||
</port>
|
||||
</output>
|
||||
</layer>
|
||||
```
|
||||
Reference in new issue
Block a user