de53c40578
* bulk change `type T` to `type *T*` in spec * update all `T` which referee to type to use *T* pattern * get back with `T` where T is dimension * fix *T*1 -> *T1* * Make italic types where was no formating
1.8 KiB
Selu
Versioned name: Selu-1
Category: Activation function
Short description: Selu is a scaled exponential linear unit element-wise activation function.
Detailed Description
Selu operation is introduced in this article, as activation function for self-normalizing neural networks (SNNs).
Selu performs element-wise activation function on a given input tensor data, based on the following mathematical formula:
\f[ Selu(x) = \lambda \left{\begin{array}{r} x \quad \mbox{if } x > 0 \ \alpha(e^{x} - 1) \quad \mbox{if } x \le 0 \end{array}\right. \f]
where α and λ correspond to inputs alpha and lambda respectively.
Another mathematical representation that may be found in other references:
\f[ Selu(x) = \lambda\cdot\big(\max(0, x) + \min(0, \alpha(e^{x}-1))\big) \f]
Attributes: Selu operation has no attributes.
Inputs
-
1:
data. A tensor of type T and arbitrary shape. Required. -
2:
alpha. 1D tensor with one element of type T. Required. -
3:
lambda. 1D tensor with one element of type T. Required.
Outputs
- 1: The result of element-wise Selu function applied to
datainput tensor. A tensor of type T and the same shape asdatainput tensor.
Types
- T: arbitrary supported floating-point type.
Example
<layer ... type="Selu">
<input>
<port id="0">
<dim>256</dim>
<dim>56</dim>
</port>
<port id="1">
<dim>1</dim>
</port>
<port id="2">
<dim>1</dim>
</port>
</input>
<output>
<port id="3">
<dim>256</dim>
<dim>56</dim>
</port>
</output>
</layer>