IntenseWebs/openvino
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Revert "add more clarity on spec and align with real implementation"

Browse Source 
This reverts commit a3b232a8fb.
This commit is contained in:
DD 2021-06-04 10:51:04 +08:00
parent 4d2f371352
commit 948ec772e0
1 changed files with 5 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/ops/activation/SoftPlus_4.md
View File

@ -19,19 +19,16 @@ SoftPlus(x) = \left\{\begin{array}{r}
\end{array}\right.
\f]
**Note**:
For numerical stability the operation reverts to the linear function when `x > threshold` where `threshold` depends on *T* and is chosen in such a way that the difference between the linear function and exact calculation is no more than `1e-6`.
The `threshold` is internally defined parameter, which can be calculated with the following formula where `alpha` is the number of digits after the decimal point, `beta` is maximum value of *T* data type:
**Note**: For numerical stability the operation reverts to the linear function when `x > threshold` where `threshold` depends on *T* and
is chosen in such a way that the difference between the linear function and exact calculation is no more than `1e-6`.
The `threshold` can be calculated with the following formula where `alpha` is the number of digits after the decimal point,
`beta` is maximum value of *T* data type:
\f[
-log(e^{10^{-\alpha}} - 1.0) < threshold < log(\beta)
\f]
For example:
If *T* is `fp32`, `threshold` range is `13.8 ~ 88.7`, in practice `threshold` is set to be `20`.
If *T* is `fp16`, in practice `threshold` is also set to be `20`，which is also ok as the data is operated in `fp32` format and then converted to `fp16` at the end.
For example, if *T* is `fp32`, `threshold` should be `20` or if *T* is `fp16`, `threshold` should be `11`.
**Attributes**: *SoftPlus* operation has no attributes.