| ActivationCube |
f(x) = x^3
|
| ActivationELU |
|
| ActivationGELU |
|
| ActivationHardSigmoid |
f(x) = min(1, max(0, 0.2*x + 0.5))
|
| ActivationHardTanH |
⎧ 1, if x > 1
f(x) = ⎨ -1, if x < -1
⎩ x, otherwise
|
| ActivationIdentity |
f(x) = x
|
| ActivationLReLU |
Leaky RELU
f(x) = max(0, x) + alpha * min(0, x)
alpha defaults to 0.01
|
| ActivationMish |
https://arxiv.org/ftp/arxiv/papers/1908/1908.08681.pdf
|
| ActivationPReLU |
|
| ActivationRationalTanh |
|
| ActivationRectifiedTanh |
|
| ActivationReLU |
f(x) = max(0, x)
|
| ActivationReLU6 |
|
| ActivationRReLU |
|
| ActivationSELU |
https://arxiv.org/pdf/1706.02515.pdf
|
| ActivationSigmoid |
f(x) = 1 / (1 + exp(-x))
|
| ActivationSoftmax |
f_i(x) = exp(x_i - shift) / sum_j exp(x_j - shift)
where shift = max_i(x_i)
|
| ActivationSoftPlus |
f(x) = log(1+e^x)
|
| ActivationSoftSign |
f_i(x) = x_i / (1+|x_i|)
|
| ActivationSwish |
f(x) = x * sigmoid(x)
|
| ActivationTanH |
f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))
|
| ActivationThresholdedReLU |
Thresholded RELU
f(x) = x for x > theta, f(x) = 0 otherwise.
|