| TanH |
\(f(x) = tanh(x) = \frac{2}{1+e^2x}-1\) |
\(f'(x) = 1 - f(x)^2\) |
\((-1,1)\) |
\(C^\infty\) |
Yes |
No |
Yes |
| SoftSign |
\(f(x)= \frac{x}{1+|x|}\) |
\(f'(x)=1-f(x)^2\) |
\((-1,1)\) |
\(C^1\) |
Yes |
No |
Yes |
| SoftPlus |
\(\displaystyle f(x)=\ln(1+e^{x})\) |
\(\displaystyle f'(x)={\frac {1}{1+e^{-x}}}\) |
\((0,\infty)\) |
\(C^\infty\) |
Yes |
Yes |
No |
| SoftExponential |
\(f(\alpha ,x) = \left\{{\begin{array}{rcl}-{\frac {\ln(1-\alpha (x+\alpha ))}{\alpha }}{\mbox{ for }}\alpha < 0\\x{\mbox{ for }}\alpha =0\\{\frac {e^{\alpha x}-1}{\alpha }}+\alpha {\mbox{ for }}\alpha > 0\end{array}}\right.\) |
\(f'(\alpha ,x)=\left\{{\begin{array}{rcl}{\frac {1}{1-\alpha (\alpha +x)}}{\mbox{ for }}\alpha <0\\e^{\alpha x}{\mbox{ for }}\alpha \geq 0\end{array}}\right.\) |
\((- \infty,\infty)\) |
\(C^\infty\) |
Yes |
Yes |
Yes iff \(\alpha = 0\) |
| Sinusoid |
\(f(x)=\sin(x)\) |
\(f'(x)=\cos(x)\) |
\([-1,1]\) |
\(C^\infty\) |
No |
No |
Yes |
| Sinc |
\(f(x)=\left\{{\begin{array}{rcl}1{\mbox{ for }}x=0\\{\frac {\sin(x)}{x}}{\mbox{ for }}x\neq 0\end{array}}\right.\) |
\(f'(x)=\left\{{\begin{array}{rcl}0{\mbox{ for }}x=0\\{\frac {\cos(x)}{x}}-{\frac {\sin(x)}{x^{2}}}{\mbox{ for }}x\neq 0\end{array}}\right.\) |
\([\approx -.217234,1]\) |
\(C^\infty\) |
No |
No |
No |
| Scaled exponential linear unit (SELU) |
\(f(\alpha ,x)=\lambda \left\{{\begin{array}{rcl}\alpha (e^{x}-1){\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) \(\lambda =1.0507\) y \(\alpha =1.67326\) |
\(f'(\alpha ,x)=\lambda \left\{{\begin{array}{rcl}f(\alpha ,x)+\alpha {\mbox{ for }}x<0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\((-\lambda \alpha,\infty)\) |
\(C^0\) |
Yes |
No |
No |
| Rectified linear unit (ReLU) |
\(f(x)=\left\{{\begin{array}{rcl}0{\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(f'(x)=\left\{{\begin{array}{rcl}0&{\mbox{for}}&x<0\\1&{\mbox{for}}&x\geq 0\end{array}}\right.\) |
\([0,\infty)\) |
\(C^0\) |
Yes |
Yes |
No |
| Randomized leaky rectified linear unit (RReLU) |
\(f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha x {\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(f'(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha {\mbox{ for }}x<0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\((-\infty, \infty)\) |
\(C^0\) |
Yes |
Yes |
No |
| Parametric rectified linear unit (PReLU) |
\(f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha x{\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(f'(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha {\mbox{ for }}x<0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\((-\infty, \infty)\) |
\(C^0\) |
Yes iff \(\alpha \geq 0\) |
Yes |
Yes iff \(\alpha = 1\) |
| Logistic (a.k.a soft step) |
\(f(x)={\frac {1}{1+e^{-x}}}\) |
\(f'(x)=f(x)(1-f(x))\) |
(0,1) |
\(C^\infty\) |
Yes |
No |
No |
| Leaky rectified linear unit (Leaky ReLU) |
\(f(x)=\left\{{\begin{array}{rcl}0.01x{\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(f'(x)=\left\{{\begin{array}{rcl}0.01{\mbox{ for }}x<0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\((-\infty, \infty)\) |
\(C^0\) |
Yes |
Yes |
No |
| Identity |
\(f(x) = x\) |
\(f''(x) = 1\) |
\((-\infty, \infty)\) |
\(C^\infty\) |
Yes |
Yes |
Yes |
| Gaussian |
\(f(x)=e^-x^2\) |
\(f'(x) = -2xe^-x^2\) |
\((0,1]\) |
\(C^\infty\) |
No |
No |
No |
| Exponential linear unit (ELU) |
\(f(\alpha ,x)=\left\{{\begin{array}{rcl}\alpha (e^{x}-1){\mbox{ for }}x<0\\x{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(f'(\alpha ,x)=\left\{{\begin{array}{rcl}f(\alpha ,x)+\alpha {\mbox{ for }}x<0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\((-\alpha,\infty)\) |
\(C^1\) when \(\alpha = 1\), otherwise \(C^0\) |
Yes iff \(\alpha \geq 0\) |
Yes iff \(0 \leq \alpha \leq 1\) |
|
| Binary step |
\(\displaystyle f(x)=\left\{{\begin{array}{rcl}0{\mbox{ for }}x <0\\1{\mbox{ for }}x\geq 0\end{array}}\right.\) |
\(\displaystyle f'(x)=\left\{{\begin{array}{rcl}0{\mbox{ for }}x\neq 0\\?{\mbox{ for }}x=0\end{array}}\right.\) |
{0,1} |
\(C^{-1}\) |
Yes |
No |
No |
| Bent Identity |
\(f(x)={\frac {{\sqrt {x^{2}+1}}-1}{2}}+x\) |
\(f'(x)={\frac {x}{2{\sqrt {x^{2}+1}}}}+1\) |
\((-\infty, \infty)\) |
\(C^\infty\) |
Yes |
Yes |
Yes |
| ArcTan |
\(f(x)=\tan ^{-1}(x)\) |
\(f'(x)={\frac {1}{x^{2}+1}}\) |
\(\left ( -\frac{\pi}{2},\frac{\pi}{2} \right )\) |
\(C^\infty\) |
Yes |
No |
Yes |
| Adaptive piecewise linear (APL) |
\(\displaystyle f(x)=\max(0,x)+\sum _{s=1}^{S}a_{i}^{s}\max(0,-x+b_{i}^{s})\) |
\(\displaystyle f'(x)=H(x)-\sum _{s=1}^{S}a_{i}^{s}H(-x+b_{i}^{s})\) |
\((-\infty, \infty)\) |
\(C^0\) |
No |
No |
No |