A catalogue of activation functions
Everything a neural network can learn beyond a straight line comes from φ, the activation function.
This index collects them: the formula, how the curve behaves, and a plot of it and its derivative. It's organised by family, from the classics every framework ships to learned and experimental functions from the research literature. A use-case guide explains which ones get picked for which jobs, and why.
374 of ≈400 functions catalogued · 18 families · 3 tiers
374 functions
The functions every framework ships: sigmoidal curves, rectifiers, thresholds and the output layers built on them.
The simplest maps: pass the input through, or snap it to a few discrete values.
Linear
Passes the input through unchanged. It's the right output for regression and appears inside residual paths. A network built only from identity activations collapses into a single linear map.
Chosen for Financial and noisy time-series forecasting, Reinforcement learning policies and value functions, Output layers: matching the target
Heaviside
Fires 1 or 0, like the original perceptron. Its derivative is zero almost everywhere, so plain gradient descent can't train it; spiking and binary networks use surrogate gradients instead.
Rosenblatt, 1958
Chosen for Binary and spiking neural networks
Signum
Binarises activations to ±1. Binary neural networks use it in the forward pass so multiplications become bit operations, and train through it with a straight-through estimator.
Courbariaux et al., 2016 (BinaryNet)
Chosen for Binary and spiking neural networks
S-shaped curves that squash the real line into a bounded interval. Their gradients vanish once they saturate.
Logistic
Squashes any input into (0, 1), so it reads as a probability. It saturates for large |x| and its gradient peaks at 0.25, which made deep sigmoid networks hard to train. Still the standard choice for binary outputs and LSTM/GRU gates.
Chosen for Medical image segmentation, Recurrent networks (LSTM, GRU), Output layers: matching the target
Hyperbolic tangent
A rescaled sigmoid (tanh x = 2σ(2x) − 1) that is zero-centred but saturating: its gradient peaks at 1 at x = 0 and vanishes for large |x|. Used for the LSTM/GRU candidate state, PPO hidden layers, GAN generator outputs and physics-informed networks.
LeCun et al., 1998 (Efficient BackProp)
Chosen for Financial and noisy time-series forecasting, Recurrent networks (LSTM, GRU), Generative adversarial networks, Physics-informed neural networks, Reinforcement learning policies and value functions, Output layers: matching the target
Scaled tanh
Tanh rescaled so that f(±1) = ±1, which keeps unit-variance inputs at roughly unit variance and keeps the units away from saturation.
LeCun et al., 1998
Inverse tangent
Shaped like tanh but approaches its limits polynomially rather than exponentially, so it saturates more gently and keeps a usable gradient further from zero.
Elliott function
A tanh-like curve with polynomial tails: it saturates more slowly than tanh, so gradients survive further from the origin. Studied as a tanh replacement in deep feed-forward nets.
Elliott, 1993; Glorot & Bengio, 2010
A piecewise-linear stand-in for the sigmoid. It's cheap and quantisation-friendly, which is why MobileNetV3 uses it in squeeze-and-excitation blocks. Some libraries use a slope of 0.2 instead of 1/6.
Chosen for Mobile, edge and quantised inference
Clips to [−1, 1] with slope 1 in between. The gradient is exactly zero outside the linear zone; binary networks use its derivative as the straight-through estimator.
Chosen for Binary and spiking neural networks
The logarithm of the sigmoid, computed stably. It mostly appears inside losses (binary cross-entropy, word2vec negative sampling) rather than as a hidden-layer activation.
erf
The Gauss error function: the normal CDF rescaled to (−1, 1). It behaves like tanh but saturates faster, because its tails are Gaussian.
Φ, Gaussian CDF
The cumulative distribution of a standard normal. As an output link it gives probit regression; inside a network it's the gate in GELU.
cloglog, Gompertz link
An asymmetric sigmoid: it leaves 0 slowly and approaches 1 quickly. It's the natural link for discrete-time survival models with proportional hazards, and for rare-event binary outcomes.
Chosen for Output layers: matching the target
gd
Connects circular and hyperbolic functions. Another smooth, odd, saturating curve between tanh and arctan in how fast it saturates.
Inverse square root unit
Inverse square root unit: tanh-shaped, but needs only a square root instead of exponentials, which is cheaper on some hardware. α sets the bounds ±1/√α (α = 1 plotted).
Carlile et al., 2017
Identity for positive inputs, something cheaper or flatter for negative ones. The default hidden-layer choice since about 2011.
Rectified linear unit
Zero for negative inputs, identity for positive ones. It's cheap, doesn't saturate for x > 0 and produces sparse activations, which made deep networks practical. A unit that only ever sees negative inputs stops learning: a 'dead' ReLU.
Nair & Hinton, 2010; Glorot et al., 2011
Chosen for Transformer feed-forward blocks, Image classification CNNs, Medical image segmentation, Mobile, edge and quantised inference, Generative adversarial networks, Implicit neural representations, Graph neural networks, Deep tabular models and deep MLPs, Reinforcement learning policies and value functions, Sparse coding and denoising
LReLU
Gives negative inputs a small slope α (0.01 by default) so units never fully die. GAN discriminators and graph attention networks typically use α = 0.2.
Maas et al., 2013
Chosen for Image classification CNNs, Medical image segmentation, Generative adversarial networks, Graph neural networks, Audio and waveform generation
Randomised leaky ReLU
A leaky ReLU whose negative slope is sampled at random during training, which adds a little regularising noise. At test time the slope is fixed to the mean, (l + u)/2 ≈ 0.23, which is what's plotted.
Xu et al., 2015
ReLU capped at 6. The bounded range keeps activations representable in low-precision fixed-point arithmetic, which is why mobile networks such as MobileNetV2 adopted it.
Krizhevsky, 2010; Howard et al., 2017
Chosen for Mobile, edge and quantised inference
Passes values only above a threshold θ (θ = 1 plotted). It was introduced for zero-bias autoencoders, where it gives sparse codes without learned biases.
Konda et al., 2015
Abs rectification
Rectifies both signs, so it discards the sign but keeps the magnitude. Early multi-stage vision systems used absolute-value rectification of filter responses.
Jarrett et al., 2009
Concatenated ReLU
ReLU(x)ReLU(−x)
Concatenates ReLU(x) and ReLU(−x), doubling the channels so no information is thrown away. It was motivated by the observation that early CNN filters tend to learn negated pairs.
Shang et al., 2016
A smooth ReLU: always positive and differentiable everywhere. Common as an output for quantities that must be positive (variances, rates), and the inner function of Mish.
Dugas et al., 2001
Chosen for Financial and noisy time-series forecasting, Physics-informed neural networks, Output layers: matching the target
Exponential linear unit
Identity for positive inputs and a smooth exponential that saturates at −α for negative ones (α = 1 plotted). The negative outputs push mean activations toward zero, which speeds up learning.
Clevert et al., 2015
Chosen for Financial and noisy time-series forecasting, Graph neural networks
Scaled ELU
A scaled ELU (λ ≈ 1.0507, α ≈ 1.6733) whose constants make activations converge to zero mean and unit variance across layers, given LeCun-normal initialisation: 'self-normalising' networks that don't need batch norm.
Klambauer et al., 2017
Chosen for Deep tabular models and deep MLPs
Continuously differentiable ELU
α = 0.5α = 1α = 2
A version of ELU that is continuously differentiable for every α. Plain ELU's derivative jumps at 0 whenever α ≠ 1. Plotted for α = 0.5, 1 and 2.
Barron, 2017
Inverse square root linear unit
ELU-shaped, but the negative side uses an inverse square root instead of an exponential, which is cheaper to compute (α = 1 plotted).
Carlile et al., 2017
ssp
Softplus shifted down by ln 2 so that f(0) = 0. SchNet uses it so that predicted molecular energies, and the forces derived from them, are smooth.
Schütt et al., 2017 (SchNet)
Chosen for Molecular and materials modelling
Functions that zero out small values, borrowed from signal denoising and sparse coding.
Zeroes inputs inside [−λ, λ] and passes the rest unchanged (λ = 0.5 plotted). This is hard thresholding, as used in wavelet denoising and sparse coding.
Chosen for Sparse coding and denoising
Soft thresholding: zero inside [−λ, λ], and larger values move toward zero by λ. It's the proximal operator of the L1 norm, which is why it appears in unrolled sparse-coding networks such as LISTA.
Chosen for Sparse coding and denoising
What's left after subtracting tanh: almost flat near zero and nearly linear for large |x|. Rarely used in practice, but available in PyTorch.
Activations that turn logits into probabilities or positive quantities. Vector functions are plotted for the two-class case.
Normalised exponential
T = 0.5T = 1T = 2
Turns a vector of logits into a probability distribution. With two classes it reduces to the sigmoid of the logit difference, which is what's plotted against x = z₁ − z₂ for temperatures T = 0.5, 1 and 2. Lower T gives a sharper distribution.
Chosen for Attention weights, Medical image segmentation, Reinforcement learning policies and value functions, Output layers: matching the target
Log of softmax, computed in one numerically stable step with log-sum-exp. It's paired with negative log-likelihood loss; most cross-entropy implementations use it internally.
Softmax of the negated inputs, so the smallest value gets the most weight. Used for soft nearest-neighbour assignments and differentiable minimums.
The Euclidean projection of the logits onto the probability simplex. Unlike softmax it can output exact zeros, which gives sparse, more interpretable attention. For two classes it's a hard sigmoid.
Martins & Astudillo, 2016
Chosen for Attention weights
Entmax
Sits between softmax (α = 1) and sparsemax (α = 2): smooth near ties, but it can still assign exact zeros. Used for sparse attention in sequence-to-sequence models.
Peters et al., 2019
Chosen for Attention weights
Maps to strictly positive values. Used as an output for rates and scale parameters (a Poisson intensity, a standard deviation); its unbounded growth makes it unstable in hidden layers.
Chosen for Output layers: matching the target
SQMAX
A softmax replacement that uses squares instead of exponentials, with c = 4. Plotted for two classes with z₂ = 0.
Wuraola et al., 2021 (via Kunc & Kléma, 2024)
Smooth self-gated units, the gated linear units inside today's Transformers, and activations whose shape is learned during training.
x multiplied by a smooth gate of itself (GELU, SiLU, Mish) and their cheaper approximations.
Gaussian error linear unit
Weights each input by the probability that a standard normal variable falls below it. It's smooth and dips slightly below zero (minimum ≈ −0.17 near x = −0.75). The default in BERT, GPT-2/3 and most Transformers before GLU variants took over.
Hendrycks & Gimpel, 2016
Chosen for Transformer feed-forward blocks, Financial and noisy time-series forecasting, Deep tabular models and deep MLPs
The fast approximation used in the original BERT and GPT-2 code. It's visually indistinguishable from exact GELU; pick whichever the pretrained weights you load were trained with.
Hendrycks & Gimpel, 2016
Chosen for Transformer feed-forward blocks, Diffusion model denoisers
Sigmoid GELU approximation
A cheaper sigmoid-based approximation of GELU. OpenAI's CLIP models were trained with it, so loading their weights means using it too.
Hendrycks & Gimpel, 2016; Radford et al., 2021 (CLIP)
Swish-1, sigmoid-weighted linear unit
The input multiplied by its own sigmoid. Smooth and non-monotonic, with a minimum of about −0.28 at x ≈ −1.28. Used in EfficientNet, the YOLO family and most diffusion-model U-Nets.
Elfwing et al., 2017; Ramachandran et al., 2017
Chosen for Image classification CNNs, Financial and noisy time-series forecasting, Diffusion model denoisers, Physics-informed neural networks, Molecular and materials modelling
h-swish
A piecewise-polynomial approximation of Swish with no exponentials, so it runs fast on mobile CPUs and quantises well. Introduced in MobileNetV3.
Howard et al., 2019
Chosen for Image classification CNNs, Mobile, edge and quantised inference
Smooth and non-monotonic, unbounded above and bounded below (minimum ≈ −0.31). Very similar to Swish with a slightly deeper negative lobe. Used in the YOLOv4 backbone.
Misra, 2019
Chosen for Image classification CNNs
An algebraic softplus: no exponentials or logarithms, so it's fast and numerically simple. b controls the curvature near zero (b = 4 plotted).
Barron, 2021
Smooth ReLU
Smooth ReLU: replaces ReLU's corner with a quadratic patch of width 2β (β = 1 plotted). Proposed to make recommender-system predictions more reproducible between training runs.
Shamir et al., 2020
ReLU²
ReLU squared. Primer's architecture search found it makes Transformer language models train more efficiently, and later efficient-LLM work adopted it.
So et al., 2021 (Primer)
Chosen for Transformer feed-forward blocks
Squared ReLU with a learnable scale and bias (initialised to s ≈ 0.894, b ≈ −0.447) that keep the output distribution normalised. Cheaper than GELU with similar accuracy in MetaFormer models.
Yu et al., 2022 (MetaFormer baselines)
Two projections, one gating the other. GLU variants are plotted with both halves tied to the same input, i.e. x · g(x).
Gated linear unit
Splits a projection into a value half a and a gate half b and multiplies a by σ(b), so the network controls how much information flows. Introduced for convolutional language models.
Dauphin et al., 2017
The gate without any nonlinearity: just the elementwise product of two projections. It was surprisingly competitive in Transformer feed-forward experiments, which suggests the multiplication matters more than the exact gate.
Shazeer, 2020
GLU with a ReLU gate. It beat standard ReLU feed-forward layers at equal compute, and FT-Transformer uses it for tabular data.
Shazeer, 2020
Chosen for Deep tabular models and deep MLPs
GLU with a GELU gate. Used in T5 v1.1, Gemma and other Transformer language models.
Shazeer, 2020
Chosen for Transformer feed-forward blocks, Diffusion model denoisers
The feed-forward activation of LLaMA, PaLM, Mistral and most open LLMs. Because it needs three weight matrices instead of two, the hidden size is usually cut to about 2/3 to keep the parameter count equal.
Shazeer, 2020
Chosen for Transformer feed-forward blocks
WaveNet gate, GTU
A bounded tanh 'content' path modulated by a sigmoid gate. The activation inside WaveNet and gated PixelCNN for audio and image generation.
van den Oord et al., 2016
Chosen for Audio and waveform generation
Activations with trainable parameters, so every neuron or channel can learn its own shape. Plots show example parameter values.
Parametric ReLU
α = 0α = 0.25α = 0.5
A leaky ReLU whose negative slope α is learned, usually one per channel and initialised to 0.25. It was used in the first network reported to beat human-level accuracy on ImageNet.
He et al., 2015
Chosen for Image classification CNNs, Medical image segmentation
Parametric Swish
β = 0.5β = 1β = 4
Swish with a trainable β that interpolates between a scaled linear function (β → 0 gives x/2) and ReLU (β → ∞). Plotted for β = 0.5, 1 and 4.
Ramachandran et al., 2017
Parametric ELU
b = 0.5b = 1b = 2
ELU with two learned positive parameters: a sets the saturation level and b the scale. Tying them as a/b keeps the function differentiable at 0. Plotted with a = 1 and b = 0.5, 1, 2.
Trottier et al., 2017
S-shaped ReLU
Three linear pieces with four learnable parameters (two thresholds, two outer slopes), so it can learn both convex and non-convex shapes. Inspired by psychophysical laws such as Weber–Fechner.
Jin et al., 2016
Adaptive piecewise linear
Adaptive piecewise linear unit: a ReLU plus a sum of learned hinge functions, so each neuron learns its own piecewise-linear shape. Plotted with one hinge (a = 0.5, b = 1).
Agostinelli et al., 2014
Outputs the maximum of k learned linear functions, so it can approximate any convex function; ReLU and leaky ReLU are special cases. It costs k times the parameters. Plotted with three example pieces.
Goodfellow et al., 2013
'Activate or not': a smooth maximum of p₁x and p₂x with learnable p₁, p₂ and switching factor β. With p₁ = 1 and p₂ = 0 it's exactly Swish; Meta-ACON predicts β from the input.
Ma et al., 2021
PAU, rational activation
A learnable rational function (a ratio of polynomials) whose denominator can't reach zero. It can closely approximate standard activations or learn new shapes. Plotted with example coefficients.
Molina et al., 2020
α = −0.5α = 0α = 0.5
One parameter α moves continuously between logarithmic (α < 0), linear (α = 0) and exponential (α > 0) behaviour, so a neuron can learn to compute sums, products or exponentials.
Godfrey & Gashler, 2015
ReLU-shaped functions with trainable slopes, thresholds or saturation levels. Plots use example or initial parameter values.
PReLU⁺
ReLU with a trainable positive slope a: the adaptive version of the sloped ReLU.
Dai et al., 2022 (via Kunc & Kléma, 2024)
MarReLU
A shifted ReLU whose floor a is set per channel from the expected value of the negative responses. Used in knowledge distillation.
Heo et al., 2019 (via Kunc & Kléma, 2024)
RPReLU
PReLU with learned horizontal (a) and vertical (b) shifts per channel. Introduced in ReActNet for binary neural networks.
Liu et al., 2020 (via Kunc & Kléma, 2024)
SAU
PReLU smoothed by convolving it with a Gaussian of learned width, which gives a closed form in terms of erf and exp.
Biswas et al., 2022 (via Kunc & Kléma, 2024)
SMU
A smooth leaky ReLU built from an erf approximation of |x|, i.e. a smooth max(x, ax). The trainable b controls how sharp the corner is.
Biswas et al., 2022 (via Kunc & Kléma, 2024)
LeLeLU
Leaky ReLU multiplied by a trainable scale a, so both slopes are learned together.
Maniatopoulos & Mitianoudis, 2021 (via Kunc & Kléma, 2024)
PREU
REU with trainable scale a and decay b, keeping a Swish-like dip for negative inputs.
Ying et al., 2019 (via Kunc & Kléma, 2024)
FReLU
ReLU with a learned vertical shift b, which lets outputs go negative and pushes the mean toward zero. (The horizontal shift is absorbed by the preceding bias.)
Qiu et al., 2018 (via Kunc & Kléma, 2024)
ShiLU
ReLU with a trainable slope a and vertical shift b per neuron.
Pishchik, 2023 (via Kunc & Kléma, 2024)
HardTanh whose slope a<sub>t</sub> grows on a schedule over training epochs while the offset b is learned, gradually turning it into a sign function for binary networks.
Liu et al., 2021 (via Kunc & Kléma, 2024)
AReLU
ReLU whose positive slope is amplified by a learned sigmoid gate and whose negative slope is a learned value clipped to [0.01, 0.99]. One pair of parameters per layer.
Chen et al., 2020 (via Kunc & Kléma, 2024)
DPReLU
PReLU with trainable slopes on both sides (initialised to 1 and 0.01). Also published as 'fully parametric ReLU'.
Balaji et al., 2020 (via Kunc & Kléma, 2024)
DPReLU plus a trainable shift m (initialised to −0.22).
Balaji et al., 2020 (via Kunc & Kléma, 2024)
PiLU
Two lines with trainable slopes meeting at a trainable point (c, c). Generalises ReLU, leaky ReLU, PReLU, DPReLU and Dual Line.
Inturrisi et al., 2021 (via Kunc & Kléma, 2024)
max(ax − b, 0)max(cx − d, 0)
Like CReLU it outputs two channels, but each has a trainable scale and threshold (scales initialised to 0.5 and −0.5).
Tang et al., 2018 (via Kunc & Kléma, 2024)
A triangular bump of trainable width a, which responds only to inputs near zero. Proposed to improve robustness to adversarial and unknown inputs.
Rozsa & Boult, 2019 (via Kunc & Kléma, 2024)
A triangle on the positive side only: it rises to a/2 and falls back to zero at a.
Wang et al., 2022 (via Kunc & Kléma, 2024)
RMAF
A Swish-like unit modelled on memristor behaviour that keeps some negative information.
Yu et al., 2020 (via Kunc & Kléma, 2024)
PTELU
Identity for positive inputs and a trainable scaled tanh for negative ones, saturating at −a.
Gupta & Duggal, 2017 (via Kunc & Kléma, 2024)
TaLU
Identity for positive inputs, tanh down to a threshold a < 0, then constant.
Mercioni & Holban, 2021 (via Kunc & Kléma, 2024)
TaLU with a second trainable threshold b above which the function becomes the identity. As written it jumps at x = b.
Mercioni & Holban, 2021 (via Kunc & Kléma, 2024)
A trainable blend of tanh and a linear term: tanh-shaped near zero, linear far away.
Shen et al., 2022 (via Kunc & Kléma, 2024)
A Mish/TanhExp relative that gates x with tanh of an ELU, with a trainable input scale a.
Mercioni & Holban, 2020 (via Kunc & Kléma, 2024)
TReLU
A special case of PTELU with a = 1: identity for positive inputs and a trainable-slope tanh for negative ones.
Zhang et al., 2019 (via Kunc & Kléma, 2024)
ReLTanh
Tanh between two trainable thresholds, continued outside them by straight lines tangent to the tanh, so the gradient never vanishes.
Wang et al., 2019 (via Kunc & Kléma, 2024)
BLU
a = −0.5a = 0.5a = 1
Interpolates between the identity (a = 0) and a smooth rectifier (a = 1) with a trainable a ∈ [−1, 1]. Being able to become the identity lets a layer effectively be skipped.
Godfrey, 2019 (via Kunc & Kléma, 2024)
ReBLU
BLU on the positive side and zero on the negative side.
Dai et al., 2022 (via Kunc & Kléma, 2024)
SiLU for negative inputs and a trainable-slope line plus a saturating term for positive ones.
Pishchik, 2023 (via Kunc & Kléma, 2024)
SC-mish, SCL-mish
Mish with its negative lobe clipped to zero. With a trainable a it is called SCL-mish (initialised to a = 0.25).
Mercioni & Holban, 2021 (via Kunc & Kléma, 2024)
SC-swish
Swish with its negative lobe clipped to zero.
Mercioni & Holban, 2022 (via Kunc & Kléma, 2024)
Swish below a trainable threshold c and the identity above it. Also proposed as R_S with a = 1.
Mercioni & Holban, 2020 (via Kunc & Kléma, 2024)
EDELU
An ELU whose switch point c is trainable, with b tied to a and c (bc = e<sup>ac</sup> − 1) so the pieces meet.
Çatalbaş & Morgül, 2023 (via Kunc & Kléma, 2024)
A learned, input-dependent blend of PReLU and PELU, one of several activation-mixing schemes compared in the same work.
Qian et al., 2018 (via Kunc & Kléma, 2024)
FELU
ELU rewritten with a power of two, which allows a fast bit-level approximation of the exponential. Mathematically equal to ELU.
Qiumei et al., 2019 (via Kunc & Kléma, 2024)
FELU with an added trainable vertical shift b.
Adem, 2022 (via Kunc & Kléma, 2024)
MPELU
ELU with trainable saturation a and steepness b; ReLU, ELU and some PELU settings are special cases.
Li et al., 2018 (via Kunc & Kléma, 2024)
A trainable mix of ReLU, ELU and a mirrored ELU, giving a flexible piecewise-smooth shape.
Jie et al., 2021 (via Kunc & Kléma, 2024)
ErfReLU
An ELU-like unit whose negative side is a trainable multiple of erf(x), saturating at −a.
Rajanand & Singh, 2023 (via Kunc & Kléma, 2024)
PSELU
SELU with its scale and saturation constants made trainable per neuron.
Pratama & Kang, 2020 (via Kunc & Kléma, 2024)
LPSELU
PSELU with a small leaky term c so negative inputs keep a gradient.
Pratama & Kang, 2020 (via Kunc & Kléma, 2024)
LPSELU_RP
LPSELU plus a trainable vertical shift m.
Pratama & Kang, 2020 (via Kunc & Kléma, 2024)
PShELU
PELU with a trainable horizontal shift c.
Grelsson & Felsberg, 2018 (via Kunc & Kléma, 2024)
PSvELU
PELU with a trainable vertical shift c.
Grelsson & Felsberg, 2018 (via Kunc & Kléma, 2024)
T-swish
Identity above a threshold c and a scaled Swish below it, with all three parameters optionally trainable.
Javid et al., 2022 (via Kunc & Kléma, 2024)
RePSU
A trainable mix of two rectified sigmoid-shaped components, generalising a shrinkage function used for image denoising.
Atto et al., 2023 (via Kunc & Kléma, 2024)
PDELU
An ELU whose exponential is replaced by a deformed (power-law) version, controlled by b, which shifts the mean output toward zero.
Cheng et al., 2020 (via Kunc & Kléma, 2024)
PFPLUS
Linear for positive inputs and a/(1 − bx)-damped for negative ones, using the Heaviside step H. With a = 0.2, b = 10 it resembles ReLU; with a = 5, b = 0.1 a linear map.
Duan et al., 2022 (via Kunc & Kléma, 2024)
PVLU
VLU with a trainable ripple amplitude a and frequency b.
Gupta & Ahuja, 2021 (via Kunc & Kléma, 2024)
PFTS
Flatted-T swish with a trainable floor T per neuron (initialised to −0.2).
Chieng et al., 2020 (via Kunc & Kléma, 2024)
PFPM
Mish on the positive side and a trainable flat level p on the negative side.
Mondal & Shrivastava, 2022 (via Kunc & Kléma, 2024)
SGT
A piecewise power function with trainable exponents on each side.
Khagi & Kwon, 2022 (via Kunc & Kléma, 2024)
s₊2L
Softplus with a trainable downward shift a ∈ [0, 1].
Vargas et al., 2023 (via Kunc & Kléma, 2024)
GReLU
A softplus in base a with a trainable sharpness b; as b grows it approaches ReLU.
Terziyan et al., 2022 (via Kunc & Kléma, 2024)
ELUs₊2L
A trainable blend of ELU and the shifted softplus s₊2L.
Vargas et al., 2023 (via Kunc & Kléma, 2024)
Simple piecewise linear and adaptive function with symmetric hinges: hinges at ±b<sub>s</sub> with separate trainable slopes on each side. Seven hinges usually worked well.
Tavakoli et al., 2021 (via Kunc & Kléma, 2024)
MeLU
PReLU plus a sum of trainable 'Mexican hat' tent functions at fixed positions, which lets each unit learn bumps in its response.
Maguolo et al., 2021 (via Kunc & Kléma, 2024)
MMeLU
A blend of one tent function and ReLU whose parameters are estimated with a Bayesian procedure.
Fakhfakh & Chaari, 2023 (via Kunc & Kléma, 2024)
GaLU
MeLU with components made of a tent followed by an inverted tent, giving alternating bumps and dips.
Nanni et al., 2020 (via Kunc & Kléma, 2024)
A scaled hard sigmoid gate with a trainable slope b; as b → ∞ it approaches ReLU. Distinct from MobileNetV3's hard swish.
Avenash & Viswanath, 2019 (via Kunc & Kléma, 2024)
An N-shaped piecewise-linear function with trainable thresholds, rising, falling and rising again. Proposed for Lipschitz-constrained networks.
Prach & Lampert, 2023 (via Kunc & Kléma, 2024)
SReLU with its thresholds fixed at 0 and 1 and trainable outer slopes. With fixed slopes it is called LiSA.
Bawa & Kumar, 2019 (via Kunc & Kléma, 2024)
All-ReLU
even layersodd layers
A leaky ReLU whose negative slope flips sign from layer to layer. Designed for sparse networks.
Curci et al., 2021 (via Kunc & Kléma, 2024)
PLU
Identity on [−b, b] and slope a outside, like a symmetric SReLU, but always invertible.
Nicolae, 2018 (via Kunc & Kléma, 2024)
AdaLU
A piecewise-linear unit with trainable offsets, slopes on each side and a saturation level e.
Mo et al., 2022 (via Kunc & Kléma, 2024)
Sigmoid- and tanh-shaped functions whose steepness, offset or bounds are learned.
One of the earliest adaptive activations: one trainable parameter controls both the slope and the amplitude of a bipolar sigmoid.
Yamada & Yabuta, 1992 (via Kunc & Kléma, 2024)
A bipolar sigmoid with trainable amplitude a and slope b.
Chen & Chang, 1996 (via Kunc & Kléma, 2024)
ASSF
The logistic sigmoid with one global trainable slope a.
Yamada & Yabuta, 1992; Nawi et al., 2009 (via Kunc & Kléma, 2024)
SVAF
Tanh with one global trainable slope a, trained with a modified backpropagation rule.
Bai, 2009 (via Kunc & Kléma, 2024)
psigmoid
A logistic sigmoid with a trainable amplitude a (per neuron or channel) and a slope b.
Ying et al., 2021; Özbay & Tezel, 2010 (via Kunc & Kléma, 2024)
PSF
A sigmoid raised to a trainable power m, which moves its steepest point and changes its asymmetry.
Chandra & Singh, 2004 (via Kunc & Kléma, 2024)
Slope- and threshold-adaptive tanh: tanh in the middle and straight lines of trainable slope b beyond ±a, so the gradient never vanishes.
Zhang et al., 2021 (via Kunc & Kléma, 2024)
GRA
A rational sigmoid-like function derived from the Riccati equation, with three trainable shape parameters.
Protonotarios et al., 2022 (via Kunc & Kléma, 2024)
GEU
The normal CDF with a global trainable width a; multiplying it by x gives PGELU.
Duan et al., 2024 (via Kunc & Kléma, 2024)
A sign function with a trainable threshold per channel. Used in ReActNet's binary networks.
Liu et al., 2020 (via Kunc & Kléma, 2024)
A trainable blend of the logistic sigmoid and a linear ramp of trainable slope b.
Jie et al., 2021 (via Kunc & Kléma, 2024)
AGumb
A trainable asymmetric sigmoid based on the Gumbel distribution (the same family as Aranda-Ordaz).
Farhadi et al., 2019 (via Kunc & Kléma, 2024)
SAAAF
The soft-root-sign shape with trainable a and b (requiring 0 < b/a < e): bounded above by a, with a dip for negative inputs.
Zhou et al., 2021 (via Kunc & Kléma, 2024)
Softsign with a trainable amplitude a and transition rate b.
Pishchik, 2023 (via Kunc & Kléma, 2024)
Swish-style units, x times a gate, with trainable gate parameters.
Softplus gated by a trainable-slope tanh, which gives a Swish-like curve with a negative lobe.
Biswas et al., 2020 (via Kunc & Kléma, 2024)
TanhExp with trainable scale b and rate c.
Biswas et al., 2020 (via Kunc & Kléma, 2024)
A softplus-like curve whose inner term is gated by tanh(ax).
Biswas et al., 2020 (via Kunc & Kléma, 2024)
AHAF
Swish with a trainable vertical scale a as well as the slope b.
Bodyanskiy & Kostiuk, 2022 (via Kunc & Kléma, 2024)
PSSiLU
SiLU whose gate is shifted down by b and renormalised, which deepens the negative region.
Dai et al., 2022 (via Kunc & Kléma, 2024)
A smooth maximum of x and bx (b initialised to 0.25): the ACON family's smooth leaky ReLU.
Ma et al., 2021 (via Kunc & Kléma, 2024)
PSGU
x gated by tanh of a scaled sigmoid; the authors also propose a matching initialisation scheme.
Li et al., 2021 (via Kunc & Kléma, 2024)
TBSReLUl
TBSReLU with a trainable scale a inside the tanh (initialised to 0.5).
Mercioni et al., 2020 (via Kunc & Kléma, 2024)
Like PSGU but with arctan and a scale a sampled at random during training.
Zheng & Wang, 2020 (via Kunc & Kléma, 2024)
AQuLU
A quadratic stand-in for SiLU: zero, then a trainable parabola, then the identity.
Wu et al., 2023 (via Kunc & Kléma, 2024)
x gated by erf of a trainable exponential.
Biswas et al., 2022 (via Kunc & Kléma, 2024)
pserf
Serf with trainable scales a and b.
Biswas et al., 2022 (via Kunc & Kléma, 2024)
PFLU with a scale a on the algebraic gate (a = 0.5 was used).
Abdool & Dear, 2023 (via Kunc & Kléma, 2024)
PGELU
GELU with a global trainable scale a, derived from noise injection (a is the RMS noise).
Duan et al., 2024 (via Kunc & Kléma, 2024)
PSTanh
A Swish-like unit built on tanh; since ½(1 + tanh(x)) = σ(2x), it is Swish with β = 2 when a = ½, b = 1.
Adu et al., 2021 (via Kunc & Kléma, 2024)
LAU
Logish with trainable per-layer parameters a and b (both initialised to 1, which gives Logish).
Zhou et al., 2023 (via Kunc & Kléma, 2024)
CosLU
The cosine counterpart of SinLU: a trainable cosine ripple is added before sigmoid gating.
Pishchik, 2023 (via Kunc & Kléma, 2024)
Swish with the sigmoid replaced by Richards' curve, a generalised logistic with extra shape parameters (the two-parameter form is plotted).
Patwardhan et al., 2018 (via Kunc & Kléma, 2024)
General recipes for learnable activations: rational functions, fractional derivatives, blends of basis functions and other parameterised families.
TAAF
Wraps any activation f in four trainable parameters that scale and shift it horizontally and vertically, which covers many published adaptive functions as special cases. Plotted for f = tanh.
Kunc et al., 2021 (via Kunc & Kléma, 2024)
Adds a trainable amplitude a and offset b to any activation g, applied per neuron. Plotted for g = tanh.
Trentin, 2001 (via Kunc & Kléma, 2024)
ARBF
A Gaussian bump with trainable centre a and width b.
Jiang et al., 2021 (via Kunc & Kléma, 2024)
LAAF
tanh(nax)σ(nax)ReLU(nax)
Gives any activation g a trainable slope a per neuron, often with a fixed scale factor n to speed up training. Developed for physics-informed networks. Plotted with n·a = 2.
Jagtap et al., 2020 (via Kunc & Kléma, 2024)
SSinH
A hyperbolic sine with trainable scales: nearly linear near zero and growing exponentially outward.
Husain et al., 2021 (via Kunc & Kléma, 2024)
SExp
A shifted exponential with trainable scales, zero at the origin.
Husain et al., 2021 (via Kunc & Kléma, 2024)
FracReLU
The a-th fractional derivative of ReLU: a = 0 gives ReLU, a = 1 the step function, and values in between interpolate. Shown for x ≥ 0.
Zamora-Esquivel et al., 2021 (via Kunc & Kléma, 2024)
UAF
A difference of softplus terms with five trainable parameters that can approximate the step, sigmoid, tanh, ReLU, leaky ReLU and Gaussian.
Yuen et al., 2021 (via Kunc & Kléma, 2024)
LEAF
Four trainable parameters around a sigmoid gate; particular settings reproduce SiLU, the sigmoid, tanh-like and ReLU-like shapes.
Bodyanskiy & Kostiuk, 2023 (via Kunc & Kléma, 2024)
MAF
A multiquadric radial basis function with trainable centre a and width b.
Huang et al., 2012 (via Kunc & Kléma, 2024)
A member of the EIS family (softplus-weighted rational functions) with trainable parameters d and e.
Biswas et al., 2021 (via Kunc & Kléma, 2024)
An EIS family member that normalises x · softplus(x) by an algebraic term, giving roughly linear growth for large inputs.
Biswas et al., 2021 (via Kunc & Kléma, 2024)
An EIS family member equivalent to a shifted Swish: d sets the offset of the gate and e its slope.
Biswas et al., 2021 (via Kunc & Kléma, 2024)
NAF
A Gaussian bump plus a sigmoid, with four network-wide trainable parameters; it performed well on small regression tasks.
Xu & Zhang, 2000 (via Kunc & Kléma, 2024)
A six-parameter trainable mix of a sine, a Gaussian and a scaled sigmoid, an extension of the scaled logistic sigmoid.
Xu & Zhang, 2002 (via Kunc & Kléma, 2024)
MWF
A Weibull-shaped bump: polynomial growth for small inputs, exponential decay for large ones, with a trainable peak position. Defined for x ≥ 0.
Husain et al., 2021 (via Kunc & Kléma, 2024)
A trainable sum of a sine and a cosine.
Efe, 2008 (via Kunc & Kléma, 2024)
CSS
A trainable sum of a sine and a sigmoid.
Özbay & Tezel, 2010 (via Kunc & Kléma, 2024)
CatAF
Mixes the identity and any activation g through a trainable angle a, so a unit can move continuously between linear and nonlinear. Plotted for g = ReLU.
Sarkar et al., 2022 (via Kunc & Kléma, 2024)
A cosine wave under a Gaussian envelope: a localised oscillation with trainable width and frequency.
Özbay & Tezel, 2010 (via Kunc & Kléma, 2024)
ABU
A trainable weighted sum of activations from a fixed pool, learned per layer, so the network can choose its own activation.
Sütfeld et al., 2020 (via Kunc & Kléma, 2024)
TCA
Averages several activations, each with its own trainable horizontal scale and shift.
Baggenstoss, 2022 (via Kunc & Kléma, 2024)
GABU
Each activation in the pool is switched on or off by a learned sigmoid gate, so every neuron can pick its activation.
Dushkoff & Ptucha, 2016 (via Kunc & Kléma, 2024)
A base activation plus trainable high-frequency sine terms, which helps physics-informed networks escape poor minima. A special case of Deep Kronecker networks.
Jagtap et al., 2022 (via Kunc & Kléma, 2024)
SLAF
A learnable polynomial per neuron. Inputs are normalised because the gradient grows with powers of x.
Goyal et al., 2019 (via Kunc & Kléma, 2024)
ChPAF
A learnable combination of Chebyshev polynomials (up to order 3 was used), shared across the network.
Deepthi et al., 2023 (via Kunc & Kléma, 2024)
LPAF
A learnable combination of Legendre polynomials, used to study how well polynomials approximate standard nonlinearities.
Venkatappareddy et al., 2021 (via Kunc & Kléma, 2024)
HPAF
A learnable combination of (physicists') Hermite polynomials.
Lokhande et al., 2020 (via Kunc & Kléma, 2024)
MoGU
A trainable mixture of Gaussian bumps and dips, motivated by the peaks and valleys that look-up-table activations tend to learn.
Wang et al., 2018 (via Kunc & Kléma, 2024)
FSA
A truncated Fourier series with trainable coefficients and base frequency (rank r = 5 was used).
Liao, 2020 (via Kunc & Kléma, 2024)
ERA
A rational activation whose denominator is a product of positive quadratics, so it has no poles and needs no absolute value, unlike PAU.
Trimmel et al., 2022 (via Kunc & Kléma, 2024)
TruG
ξ = [0, ∞)ξ = [−1, 1]
The mean of a Gaussian truncated to [ξ₁, ξ₂]: [0, ∞) gives a smooth ReLU and [−1, 1] a tanh-like curve. The truncation points can be trained.
Su et al., 2017 (via Kunc & Kléma, 2024)
KAF
A non-parametric activation: a trainable weighted sum of Gaussian kernels centred on a fixed dictionary of points.
Scardapane et al., 2019 (via Kunc & Kléma, 2024)
A Laplacian-kernel bump with trainable sharpness and centre; one of the activations trained in random-weight networks.
Ertuğrul, 2018 (via Kunc & Kléma, 2024)
Periodic and radial functions, the many descendants of Swish and Mish, and other proposals from the literature that are not yet mainstream.
Oscillating and bump-shaped functions used for implicit representations, periodic signals and RBF networks.
Sinusoid
A periodic activation for coordinate-based networks. With ω₀ = 30 and a matching initialisation, SIRENs represent images, audio and signed distance fields in fine detail, and their derivatives are SIRENs too. Plotted with ω₀ = 1.
Sitzmann et al., 2020 (SIREN)
Chosen for Implicit neural representations, Physics-informed neural networks
Periodic and even. Appears in random Fourier-feature networks and as an alternative to sine in implicit neural representations.
A decaying oscillation whose Fourier transform is a box: an ideal low-pass filter in function form. Studied as an activation for function approximation.
The identity plus a periodic ripple, which gives networks an inductive bias toward periodic signals while staying monotonic. BigVGAN uses a version with learnable a for audio synthesis. Plotted with a = 1.
Ziyin et al., 2020
Chosen for Audio and waveform generation
GCU
Oscillates with growing amplitude. Because it is non-monotonic with many zeros, a single GCU neuron can learn XOR.
Noel et al., 2021
Sinu-sigmoidal linear unit
A Swish-like gated unit that adds a learned sinusoidal ripple to the input before gating. Plotted with a = b = 1.
Paul et al., 2022
RBF
A bump that responds only near zero. The classic activation of radial basis function networks, and also used in implicit neural representations.
Chosen for Implicit neural representations
Mexican hat
The 'Mexican hat': a negated second derivative of a Gaussian. Wavelet neural networks use it to capture localised, multi-scale features.
A radial basis function that grows with distance from the centre (c = 1 plotted). Used in RBF interpolation and some RBF networks.
A heavy-tailed bump: it decays like 1/|x| rather than exponentially, so each unit responds to a wider neighbourhood than a Gaussian (c = 1 plotted).
PMAF
Two Mexican-hat wavelets centred at ±a (C = 2π<sup>−1/4</sup>/√3), so the unit responds to inputs near a or −a.
Liu et al., 2016 (via Kunc & Kléma, 2024)
PRBF
Three Gaussian bumps centred at −2a, 0 and 2a, so the unit responds to several separate input ranges.
Liu et al., 2016 (via Kunc & Kléma, 2024)
A sine with reduced amplitude and an upward offset; frequencies a from 0.2 to 4 were compared.
Gashler & Ashmore, 2014 (via Kunc & Kléma, 2024)
A non-negative cosine bump that is zero at the origin.
Daskin, 2018 (via Kunc & Kléma, 2024)
A decreasing line with a cosine ripple on top.
Xu & Philip Chen, 2020 (via Kunc & Kléma, 2024)
A sine ripple on a decreasing line; slopes a = 1, 1.5 and 2 were used.
Chan et al., 2018 (via Kunc & Kléma, 2024)
ASU
The sine counterpart of the growing cosine unit: an even function oscillating with growing amplitude.
Rahman et al., 2023 (via Kunc & Kléma, 2024)
SSU
A sinc shifted right by π and scaled by π, so its main lobe sits at x = π.
Gustineli, 2022 (via Kunc & Kléma, 2024)
DSU
The difference of two shifted sincs: an odd, decaying oscillation.
Gustineli, 2022 (via Kunc & Kléma, 2024)
A Mexican-hat shape with a single width parameter; a = ½ gives the Ricker wavelet.
Efe, 2008 (via Kunc & Kléma, 2024)
Reshaped, shifted, cascaded and piecewise versions of the logistic sigmoid and tanh.
SSS
The logistic sigmoid stretched by a and shifted by b, so the transition can be placed and sharpened for a particular input scale.
Arai & Imamura, 2018 (via Kunc & Kléma, 2024)
VSF
An early three-parameter sigmoid: a scales the output, b sets the slope and c shifts it down, which can make it zero-centred.
Han & Moraga, 1995 (via Kunc & Kléma, 2024)
The average of two shifted sigmoids, so the curve rises in two steps and its derivative has two peaks.
Sodhi & Chandra, 2014 (via Kunc & Kléma, 2024)
A sigmoid applied to an algebraic squashing of the input. Because the inner term is itself bounded, the output never gets close to 0 or 1.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
TS-sigmoid
A sigmoid multiplied by a sum of three shifted sigmoids, so it climbs to 3 through a staircase of intermediate levels.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
A sigmoid in the middle with straight lines of slope a outside ±b, so the gradient never vanishes. The slope a should be larger than the sigmoid's own slope at b.
Qin et al., 2019 (via Kunc & Kléma, 2024)
Sigmoid + linear
A sigmoid plus a small linear term, which keeps a non-zero gradient everywhere.
Roodschild et al., 2020 (via Kunc & Kléma, 2024)
SRS
Smooth, non-monotonic and bounded: positive inputs saturate at a, while negative inputs dip below zero and then return to it.
Li & Zhou, 2020 (via Kunc & Kléma, 2024)
SC
A smooth version of clipping to [0, 1]: nearly linear between 0 and 1 and flat outside, with a controlling how sharp the corners are.
Klimek & Perelstein, 2020 (via Kunc & Kléma, 2024)
Two exponential pieces that saturate at a and −c, giving a tanh-like curve whose gradient can be scaled separately on each side.
Kong & Takatsuka, 2017 (via Kunc & Kléma, 2024)
A cubic sigmoid that is exactly 0 below −a/2 and exactly 1 above a/2, so it can switch paths off completely. Used for conditional computation in differentiable tree ensembles.
Hazimeh et al., 2020 (via Kunc & Kléma, 2024)
Softsign rescaled to (0, 1): a sigmoid substitute that needs no exponential, which mattered on early hardware.
Elliott, 1993 (via Kunc & Kléma, 2024)
The unnormalised sinc applied to a sigmoid. Since σ(x) stays in (0, 1), the result is a gentle decreasing curve from 1 toward sin(1).
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
SG
The product of the logistic and Gumbel CDFs: a sigmoid that rises later and more steeply than σ(x).
Kaytan et al., 2022 (via Kunc & Kléma, 2024)
A tanh-like curve that saturates at ±1/√2.
Kumar & Sodhi, 2023 (via Kunc & Kléma, 2024)
A base-√2 relative of NewSigmoid that saturates at ±1/4. The survey notes the original paper's formula appears to contain a typo; this is the survey's corrected form.
Kumar & Sodhi, 2023 (via Kunc & Kléma, 2024)
The Gumbel CDF used as a sigmoid: asymmetric, leaving 0 quickly and approaching 1 slowly (the mirror image of cloglog). Used in networks for financial time-series forecasting.
Gomes et al., 2010 (via Kunc & Kléma, 2024)
A Swish-like gated unit whose gated input gets a bump from sech(x) near zero.
Közkurt et al., 2023 (via Kunc & Kléma, 2024)
pSechSig
SechSig with a fixed parameter a that sets both the height and the position of the bump.
Közkurt et al., 2023 (via Kunc & Kléma, 2024)
Like SechSig but adds tanh(x) to the gated input, steepening the curve around zero.
Közkurt et al., 2023 (via Kunc & Kléma, 2024)
pTanhSig
TanhSig with a fixed parameter a that scales and shifts the tanh term.
Közkurt et al., 2023 (via Kunc & Kléma, 2024)
MSAF
Sums shifted sigmoids so a neuron has several stable output levels instead of two. Plotted as the two-step example −1 + σ(x) + σ(x + 4).
Cai et al., 2015 (via Kunc & Kléma, 2024)
An algebraic sigmoid that saturates at ±1 without exponentials, approaching its limits polynomially.
Duch & Jankowski, 1999 (via Kunc & Kléma, 2024)
{σ₂, tanh}{tanh, 0}{σ₂, ax}
A family that uses one function for positive inputs and another for negative ones, mixing the bipolar sigmoid σ₂(x) = 2σ(x) − 1, tanh, zero and a linear piece. Three of the proposed pairs are plotted.
Guevraa et al., 2021 (via Kunc & Kléma, 2024)
MEF
An algebraic sigmoid (ISRU with a = 1) shifted up by one half.
Burhani et al., 2015 (via Kunc & Kléma, 2024)
An asymmetric link function from statistics that includes the logistic (a = 1) and complementary log-log (a → 0) curves as special cases.
Gomes & Ludermir, 2013; Aranda-Ordaz, 1981 (via Kunc & Kléma, 2024)
Fixed (non-learned) modifications of ReLU: different negative branches, caps, shifts and smoothings.
ReLU moved down by one so the floor is at −1, which brings mean activations closer to zero.
Clevert et al., 2016 (via Kunc & Kléma, 2024)
SlReLU
ReLU with a steeper positive slope a (its authors recommend 1 to 10), the mirror image of Leaky ReLU's idea.
Seo et al., 2017 (via Kunc & Kléma, 2024)
Identity for positive inputs and a small oscillation for negative ones, so negative inputs keep a non-zero gradient.
Gupta & Ahuja, 2021; Rodrigues, 2018 (via Kunc & Kléma, 2024)
The identity for negative inputs and a sine for positive ones, so positive outputs stay bounded by 1.
Xu & Philip Chen, 2020 (via Kunc & Kléma, 2024)
VLU
ReLU with a sinusoidal ripple added everywhere, so even negative inputs have a gradient.
Gupta & Ahuja, 2021 (via Kunc & Kléma, 2024)
NLReLU
ReLU with a logarithmic positive side, which compresses large activations.
Liu et al., 2019 (via Kunc & Kléma, 2024)
SLU
Identity for positive inputs and a scaled, shifted softplus for negative ones, with parameters chosen so the function is continuous and differentiable at zero and negative outputs pull the mean toward zero.
Zhao et al., 2017 (via Kunc & Kléma, 2024)
ReSP
Softplus for negative inputs and a steeper line for positive ones; slopes between 1.4 and 2.0 were found to work well.
Xu et al., 2018 (via Kunc & Kléma, 2024)
PReNU
ReLU whose positive side bends slightly below the identity, making the positive part nonlinear.
Jaafari et al., 2020 (via Kunc & Kléma, 2024)
BReLU, ReLUN
ReLU capped at a, which prevents unbounded activations; ReLU6 is the special case a = 6. Also published independently as ReLUN.
Liew et al., 2016 (via Kunc & Kléma, 2024)
SvHardTanh
HardTanh moved up by a, so its output is centred at a instead of zero.
Kim et al., 2021 (via Kunc & Kléma, 2024)
ShHardTanh
HardTanh moved left by a, so its linear zone is no longer centred on zero.
Kim et al., 2021 (via Kunc & Kléma, 2024)
BLReLU
Leaky ReLU whose positive side also flattens to slope a above the threshold b, with c = (1 − a)b keeping it continuous.
Liew et al., 2016 (via Kunc & Kléma, 2024)
Zero inside [−a, a] and rising linearly outside it on both sides: soft thresholding of the absolute value.
Pan et al., 2023 (via Kunc & Kléma, 2024)
AbsLU
Identity for positive inputs and a scaled absolute value for negative ones, so negative inputs also produce positive outputs.
Alkhouly et al., 2021 (via Kunc & Kléma, 2024)
mReLU
A triangular 'tent' that peaks at 1 for x = 0 and is zero outside [−1, 1], suppressing unusual (large) inputs.
Zhao & Griffin, 2016 (via Kunc & Kléma, 2024)
LSPTLU, LRTLU
A leaky ReLU that peaks at a and falls back to zero by 2a, so only a band of positive inputs is passed strongly.
Shan et al., 2019 (via Kunc & Kléma, 2024)
A smooth stand-in for |x|: a quadratic-like patch near zero that joins the absolute value at ±1.
Vallés-Pérez et al., 2023 (via Kunc & Kléma, 2024)
A tanh-smoothed absolute value; a = 1 gives LiSHT and smaller a makes it closer to |x|. Its authors used a = 0.01.
Vallés-Pérez et al., 2023 (via Kunc & Kléma, 2024)
DLU
ReLU for positive inputs and a softsign for negative ones, which saturates at −a instead of being zero. Also published as DLU.
Lin & Shen, 2018 (via Kunc & Kléma, 2024)
ReLU plus a linear term ax, equivalent to a leaky ReLU whose positive slope is 1 + a.
Elakkiya & Dejey, 2024 (via Kunc & Kléma, 2024)
NCReLU
ReLU(x)−ReLU(−x)
Like CReLU, it doubles each channel, but keeps the negative part's sign, so the two outputs are max(x, 0) and min(x, 0).
Zagoruyko & Komodakis, 2017 (via Kunc & Kléma, 2024)
DRLU
ReLU shifted right by a fixed delay a, so it only switches on once the input exceeds a.
Shan et al., 2022 (via Kunc & Kléma, 2024)
DisReLU
ReLU whose corner is moved to (−a, −a); a = 1 gives the shifted ReLU.
Macêdo et al., 2019 (via Kunc & Kléma, 2024)
MLReLU
A translated leaky ReLU in which one parameter a sets both the threshold and the negative slope. As written, the function jumps at x = −a.
Yang et al., 2022 (via Kunc & Kléma, 2024)
FTS
Swish on the positive side and a flat value T on the negative side (T = −0.2 is recommended); T = 0 was also published as ReLU-Swish.
Chieng et al., 2018 (via Kunc & Kléma, 2024)
OAF
The sum of ReLU and SiLU: Swish's negative lobe with a steeper positive slope.
Sharma, 2020 (via Kunc & Kléma, 2024)
REU
Identity for positive inputs and x·eˣ for negative ones, which gives Swish's non-monotonic dip with ReLU's positive side.
Ying et al., 2019 (via Kunc & Kléma, 2024)
ADA
A biologically inspired unit that fires most strongly just above zero and decays for larger inputs, like an apical dendrite's response.
Georgescu et al., 2023 (via Kunc & Kléma, 2024)
LADA
ADA with a small leaky slope c for negative inputs.
Georgescu et al., 2023 (via Kunc & Kléma, 2024)
An ELU alternative whose negative side is tanh(x), saturating at −1.
Guevraa et al., 2021 (via Kunc & Kléma, 2024)
Swish and ReLU activation
Swish-like on the negative side and identity on the positive side. (The survey reconstructs the formula; the original paper's version was malformed.)
Qureshi & Umar, 2022 (via Kunc & Kléma, 2024)
The larger of x and the sigmoid: sigmoid-shaped for negative inputs, identity once x overtakes σ(x).
Xu & Philip Chen, 2020 (via Kunc & Kléma, 2024)
ThLU
Identity for positive inputs and a half-rate tanh for negative ones, saturating at −1.
Fu et al., 2022 (via Kunc & Kléma, 2024)
DiffELU
An ELU variant whose negative side is a difference of exponentials, giving a dip and a slight negative tail.
Hu et al., 2020 (via Kunc & Kléma, 2024)
PolyLU, FPLUS
An ELU look-alike whose negative side is rational rather than exponential, saturating at −1. Also published as FPLUS.
Feng & Yang, 2023 (via Kunc & Kléma, 2024)
IpLU
Identity for positive inputs; negative inputs are damped by 1/(1 + |x|<sup>a</sup>), so their slope stays small.
Alkhouly et al., 2021 (via Kunc & Kléma, 2024)
PoLU
An ELU-like unit with a power-law negative side that saturates at −1; a = 1, 1.5 and 2 were tested.
Li et al., 2018 (via Kunc & Kléma, 2024)
PFLU
A Swish-like gated unit whose gate is the algebraic sigmoid ½(1 + x/√(1 + x²)), with no exponentials.
Zhu et al., 2021 (via Kunc & Kléma, 2024)
FPFLU
A cheaper PFLU variant: identity for positive inputs and a curve that returns toward −1 for negative ones.
Zhu et al., 2021 (via Kunc & Kléma, 2024)
LSELU
SELU with an extra leaky term c on the negative side, so it no longer saturates completely.
Chen et al., 2021 (via Kunc & Kléma, 2024)
SERLU
A SELU-style unit whose negative side is x·eˣ, so it bumps down and returns to zero instead of saturating.
Zhang & Li, 2018 (via Kunc & Kléma, 2024)
sSELU
SELU with the negative inputs rescaled by c before the exponential, which slows its saturation.
Chen et al., 2021 (via Kunc & Kléma, 2024)
ELU-like for negative inputs, identity on [0, 1], and steeper above 1 thanks to an added sigmoid-weighted term. a = 0 gives ReLU.
Kiliçarslan & Celik, 2021 (via Kunc & Kléma, 2024)
RSigELU with separate parameters for the positive (a) and negative (b) sides.
Kiliçarslan & Celik, 2021 (via Kunc & Kléma, 2024)
ELU on the negative side and an accelerating positive side built from a hard sigmoid, cheap to compute.
Kiliçarslan, 2022 (via Kunc & Kléma, 2024)
Softsign for negative inputs, identity up to b, then a logarithmic positive side that compresses large activations.
Wang et al., 2020 (via Kunc & Kléma, 2024)
PSoftplus
Softplus with a scale a and a shift b; the shift lets the mean output move toward zero.
Sun et al., 2019 (via Kunc & Kléma, 2024)
Soft++
A non-saturating softplus extension: an added linear term keeps negative inputs from flattening out. a = 1, b = 2 worked well.
Ciuparu et al., 2020 (via Kunc & Kléma, 2024)
bfire
A smoothed absolute value: a parabola near zero joined to straight lines, so it fires for both large positive and large negative inputs.
Li et al., 2013 (via Kunc & Kléma, 2024)
bbfire
Bi-firing capped at b on both sides, giving an inverted bell shape.
Liew et al., 2016 (via Kunc & Kléma, 2024)
ShELU
ELU moved left by a fixed shift b.
Grelsson & Felsberg, 2018 (via Kunc & Kléma, 2024)
SvELU
ELU moved down by a fixed shift b.
Grelsson & Felsberg, 2018 (via Kunc & Kléma, 2024)
Variations on self-gating proposed since 2018, mostly evaluated on image classification benchmarks.
Swish multiplied by a constant β between 1 and 2 (β = 1.375 plotted), which its author found trained faster on image classification.
Alcaide, 2018
Exponential linear sigmoid squashing
Swish for positive inputs and an ELU-style exponential times the sigmoid for negative ones. Found with a genetic search over activation shapes.
Basirat & Roth, 2018
The hard-sigmoid version of ELiSH, cheaper to compute.
Basirat & Roth, 2018
Close to linear for positive inputs, because tanh(eˣ) reaches 1 quickly, with a small negative lobe. Proposed for lightweight networks.
Liu & Di, 2020
Log-softplus error function: Mish's shape with erf in place of tanh. Reported gains over Mish and Swish on image benchmarks.
Nag & Bhattacharyya, 2021
Gates the input with the logarithm of 1 + σ(x); a gentler gate than Swish's, saturating at ln 2.
Zhu et al., 2021
Combines ideas from Mish and Logish: tanh of the log-sigmoid gate.
Wang et al., 2022
Linearly scaled hyperbolic tangent
Even and non-negative: negative inputs map to positive outputs. Its authors argue the large gradients on the negative side avoid dying neurons.
Roy et al., 2019
SGELU
A symmetric (even) variant of GELU: negative inputs give positive outputs, mirroring the positive side. Its authors report more symmetrically distributed weights.
Yu & Su, 2019 (via Kunc & Kléma, 2024)
CaLU
GELU's idea with the Cauchy CDF as the gate. The heavy-tailed gate opens slowly, so the negative lobe is wide and keeps growing in magnitude.
Wu et al., 2023 (via Kunc & Kléma, 2024)
LaLU
GELU's idea with the Laplace CDF as the gate, which has exponential tails like the logistic but a sharper peak in its density.
Wu et al., 2023 (via Kunc & Kléma, 2024)
CoLU
A self-gated unit whose gate is built from a double exponential, collapsing negative inputs toward zero.
Vagerwal, 2021 (via Kunc & Kléma, 2024)
TS-swish
SiLU multiplied by a sum of three shifted sigmoids, so the slope increases in stages as the input grows.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
Gates the input with the sigmoid of e<sup>−x</sup>: the gate is about 1 for negative inputs and falls to ½ for large positive ones.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
Despite the name, this simplifies algebraically to σ(−x): a decreasing sigmoid.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
The sigmoid's own derivative used as an activation: a bell-shaped bump peaking at 0.25, similar to a Gaussian.
Koçak & Üstündağ Şiray, 2021 (via Kunc & Kléma, 2024)
A self-gated unit whose gate rises from 0 to ln 2. Its authors report it beating Mish, Logish and Smish on MNIST and CIFAR-10.
Kaytan et al., 2023 (via Kunc & Kléma, 2024)
x gated by the complementary log-log curve, which opens late and then quickly.
Wu et al., 2023 (via Kunc & Kléma, 2024)
x gated by the LogLog (Gumbel CDF) curve, which opens early and then slowly.
Wu et al., 2023 (via Kunc & Kléma, 2024)
Even and non-negative, like LiSHT but built from arctan: both large positive and large negative inputs give large positive outputs.
Tümer-Sivri et al., 2023 (via Kunc & Kléma, 2024)
pLogish
Logish with fixed scales a and b; a large b makes the gate switch sharply at zero.
Zhu et al., 2021 (via Kunc & Kléma, 2024)
Gates x with tanh of GELU. Its author found it performed on a par with Mish and Swish.
Naveen, 2022 (via Kunc & Kléma, 2024)
Identity for positive inputs; for negative inputs it follows x·e^{x}, which dips and then returns toward zero, like a sharper Swish.
Xu & Philip Chen, 2020 (via Kunc & Kléma, 2024)
TSReLU
Mish's recipe with the sigmoid inside tanh. Because tanh(σ(x)) never exceeds tanh(1) ≈ 0.76, the positive slope is below 1.
Mercioni et al., 2020 (via Kunc & Kléma, 2024)
TBSReLU
TSReLU with the bipolar sigmoid tanh(x/2) inside, so the gate is negative for negative inputs and the output is non-negative on both sides.
Mercioni et al., 2020 (via Kunc & Kléma, 2024)
Derivative of SiLU
The derivative of SiLU used as an activation: a sigmoid-like curve that overshoots to about 1.1 and dips to about −0.1. Proposed for reinforcement-learning value networks.
Elfwing et al., 2018 (via Kunc & Kléma, 2024)
SiLU whose gate is itself driven by SiLU, which flattens the negative lobe.
Verma et al., 2023 (via Kunc & Kléma, 2024)
MSiLU
SiLU plus a small Gaussian bump at the origin, which its authors found speeds up convergence.
Hayou et al., 2018 (via Kunc & Kléma, 2024)
TSiLU
SiLU squashed by tanh, which bounds the output at 1. (The survey's printed formula has a typo in the denominator; this is the intended function.)
Verma et al., 2023 (via Kunc & Kléma, 2024)
ASiLU
SiLU passed through arctan, which bounds it at π/2 but approaches the bound slowly.
Verma et al., 2023 (via Kunc & Kléma, 2024)
x gated by the sigmoid of arctan(x). The gate stays between σ(−π/2) ≈ 0.17 and σ(π/2) ≈ 0.83, so both sides grow linearly.
Verma et al., 2023 (via Kunc & Kléma, 2024)
Odd and smooth, peaking near x ≈ 1.2 and decaying back to zero at both ends, so large inputs of either sign are suppressed.
Njikam & Zhao, 2016 (via Kunc & Kléma, 2024)
Efficient asymmetric nonlinear activation: x · tanh(softplus(x)/2) simplified to a cheap closed form. Very close to Swish.
Chai et al., 2022 (via Kunc & Kléma, 2024)
A self-gated unit whose gate is a sine of the sigmoid. Its authors report it scaling better than Swish and Mish as networks get deeper.
Douge et al., 2021 (via Kunc & Kléma, 2024)
The tanh approximation of GELU rewritten with a sigmoid (since tanh(u) = 2σ(2u) − 1), so it equals GELU's tanh form exactly.
Ahmed et al., 2023 (via Kunc & Kléma, 2024)
Activations built from powers, roots, logarithms and exponentials rather than sigmoids or rectifiers.
RePU, power activation
a = 0.5a = 2a = 3
ReLU raised to a power a: a = 0 gives the step function, a = 1 ReLU, a = 2 the squared ReLU. Also proposed earlier as a 'power activation'.
Berradi, 2018; Li et al., 2019 (via Kunc & Kléma, 2024)
AppReLU
RePU with an extra scale a.
Saha et al., 2020 (via Kunc & Kléma, 2024)
EvenPowLin
An even function: a power curve near zero joined to straight lines, so it grows linearly on both sides.
Nasiri & Ghiasi-Shirazi, 2021 (via Kunc & Kléma, 2024)
OddPowLin
The odd counterpart of EPLAF, as printed in the survey: a power curve near zero (negated for negative inputs) joined to straight lines. With these outer pieces both sides end up rising linearly.
Nasiri & Ghiasi-Shirazi, 2021 (via Kunc & Kléma, 2024)
SQLU
An ELU-like unit built from squares instead of exponentials: SQNL's negative half joined to the identity. Designed for low-power hardware.
Wuraola et al., 2021 (via Kunc & Kléma, 2024)
Square swish
A Swish-inspired curve made of quadratic pieces, cheap on hardware without exponentials.
Wuraola et al., 2021 (via Kunc & Kléma, 2024)
SqREU
A square-based stand-in for REU: identity for positive inputs and a quadratic dip for negative ones, with no exponentials.
Wuraola et al., 2021 (via Kunc & Kléma, 2024)
SqSoftplus
A quadratic-patch replacement for softplus: zero, then a parabola, then the identity, so it is differentiable without exponentials.
Wuraola et al., 2021 (via Kunc & Kléma, 2024)
LogSQNL
SQNL rescaled to (0, 1): a logistic-sigmoid replacement built from two parabolas.
Wuraola & Patel, 2018 (via Kunc & Kléma, 2024)
SQRT
A signed square root: unbounded but sub-linear, so it compresses large inputs while keeping their sign. Reported to beat tanh and ReLU on CIFAR-10.
Yang et al., 2018 (via Kunc & Kléma, 2024)
SSAF
The signed square root with a scale parameter a.
Khachumov et al., 2023 (via Kunc & Kléma, 2024)
A quadratic polynomial of the softsign: bounded, non-monotonic and asymmetric, approaching 1 for large positive inputs and 0 for large negative ones.
Mishra et al., 2021 (via Kunc & Kléma, 2024)
SBAF
Proposed for classifying the habitability of exoplanets; defined on [0, 1], where it dips in the middle and equals 1 at both ends.
Saha et al., 2018 (via Kunc & Kléma, 2024)
LAF, symlog
A signed logarithm that compresses large values in both directions. Proposed early as the LAF and later as symlog, which DreamerV3 uses to squash rewards and values.
Bilski, 2000; Hafner et al., 2023 (via Kunc & Kléma, 2024)
The inverse of symlog: expands values symmetrically. DreamerV3 uses it to decode predictions made in symlog space.
Hafner et al., 2023 (via Kunc & Kléma, 2024)
Scaled polynomial constant unit: linear for negative inputs, a high-order polynomial in the middle and constant beyond d. Designed for self-normalising networks.
Kiseľák et al., 2020 (via Kunc & Kléma, 2024)
PUAF
A polynomial family that can approximate the sigmoid, ReLU and Swish by choosing a, b and c (a = 1, b = 0, c = 0 gives ReLU).
Hwang & Kim, 2023 (via Kunc & Kléma, 2024)
Found by an evolutionary search: steep near zero, flattening a little, then growing exponentially. Its finders report it beating ReLU and tanh in LSTMs.
Vijayaprabakaran & Sathiyamurthy, 2022 (via Kunc & Kléma, 2024)
m-arcsinh
A scaled product of arcsinh and the square root of |x|; usable both as a network activation and as a support-vector-machine kernel.
Parisi, 2020 (via Kunc & Kléma, 2024)
A scaled hyperbolic sine for positive inputs and a cubic for negative ones.
Parisi et al., 2021 (via Kunc & Kléma, 2024)
Squared arctan minus the identity: a mostly decreasing curve with a bump near zero.
Xu & Philip Chen, 2020 (via Kunc & Kléma, 2024)
Hyperbolic cosine linearised squashing function: log-cosh based, close to ReLU but smooth, with a slight negative lobe.
Abdel-Nabi et al., 2023 (via Kunc & Kléma, 2024)
A quadratic multiplied by a Gaussian envelope, so it responds only near zero.
Efe, 2008 (via Kunc & Kléma, 2024)
A decreasing exponential used as an activation in a physics-oriented network.
Pan et al., 2020 (via Kunc & Kléma, 2024)
An exponentially weighted tanh: a small negative dip for negative inputs and explosive growth for positive ones.
Kalaiselvi et al., 2022 (via Kunc & Kléma, 2024)
Found by neuroevolution as the best recurrent activation for LSTMs on a human-activity dataset: ReLU plus a saturating bump.
Vijayaprabakaran & Sathiyamurthy, 2021 (via Kunc & Kléma, 2024)
NCU
A cubic that rises near zero and falls beyond ±1/√3, making it strongly non-monotonic.
Gustineli, 2022 (via Kunc & Kléma, 2024)
SQU
A parabola with its minimum at x = −½; simple and non-monotonic.
Gustineli, 2022 (via Kunc & Kléma, 2024)
A smoothed step function built with the mollified absolute value; its odd counterpart x/√(x² + ε) is called BipolarPlus.
Pan et al., 2023 (via Kunc & Kléma, 2024)
SMU-1
A smoothed leaky ReLU; also published as SMU-1.
Pan et al., 2023; Biswas et al., 2022 (via Kunc & Kléma, 2024)
STFPlus
A smoothed soft-shrink: flat near zero and shifted linear beyond ±a, with rounded corners.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A smoothed pan function: near zero inside [−a, a], rising linearly outside on both sides.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A smoothed ReLU capped at 1.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A smoothed symmetric SReLU: slope 1 near zero and slope a beyond ±t.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A smoothed HardTanh with rounded corners at ±1.
Pan et al., 2023 (via Kunc & Kléma, 2024)
HTFPlus
A smoothed hard-shrink: x multiplied by a smooth window that is near zero inside [−a, a].
Pan et al., 2023 (via Kunc & Kléma, 2024)
The MSRF smoothing applied to ELU's two pieces, as written in the original. Note that, unlike ELU, this form stays positive.
Pan et al., 2023 (via Kunc & Kléma, 2024)
Swish with the sigmoid replaced by the algebraic StepPlus: no exponentials, same overall shape.
Pan et al., 2023 (via Kunc & Kléma, 2024)
An algebraic Mish-like function built from nested BipolarPlus terms, as written in the original.
Pan et al., 2023 (via Kunc & Kléma, 2024)
Logish with the sigmoid replaced by StepPlus.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A softsign whose |x| is mollified, making it smooth at zero. With ε = 1 it coincides with rootsig.
Pan et al., 2023 (via Kunc & Kléma, 2024)
A smoothed approximation of SignReLU (DLU): ReLU-like for positive inputs and a softsign-like dip for negative ones.
Pan et al., 2023 (via Kunc & Kléma, 2024)
One-off proposals and specialised activations that are worth knowing about.
Almost linear with a gentle bend: the slope goes from 0.5 for large negative inputs to 1.5 for large positive ones. Smooth, monotonic and unbounded.
Square nonlinearity: a tanh-shaped curve built from two quadratic pieces, cheap enough for hardware without exponential units.
Wuraola & Patel, 2018
Tanh with the negative side scaled down (a = 0.25), the saturating analogue of a leaky ReLU. A broad NLP comparison later found it unusually stable across tasks.
Xu et al., 2016; Eger et al., 2018
Used in a neural dependency parser to model three-way interactions between word, part-of-speech and label features directly. Unbounded and fast-growing, so it needs careful scaling.
Chen & Manning, 2014
No function matches those filters.
Every catalogued function on one page. Click a name to jump to its card.
| Function | Curve | Family | Range | Smooth | Monotonic | PyTorch |
|---|---|---|---|---|---|---|
| 1 − cosine | Periodic & radial | [0, 2] | ✓ | – | – | |
| 1.5-entmax | Output & normalising | [0, 1], sums to 1 | ✓ | ✓ | – | |
| Absolute linear unit | Rectifier variants | ≈ [0, ∞) | – | – | – | |
| Absolute value | Rectifiers (ReLU family) | [0, ∞) | – | – | – | |
| ACON-B | Adaptive self-gated | (−∞, ∞) | ✓ | ✓ | – | |
| ACON-C | Learned & adaptive | learned | ✓ | ✓ | – | |
| Adaptive blending unit | Rational, fractional & other adaptive families | (−∞, ∞) | – | ✓ | – | |
| Adaptive Gumbel | Adaptive sigmoidal | ≈ [0, 1] | ✓ | ✓ | – | |
| Adaptive HardTanh | Adaptive rectifiers | ≈ [−1, 1] | – | ✓ | – | |
| Adaptive hybrid activation function | Adaptive self-gated | ≈ [−0.42, ∞) | ✓ | – | – | |
| Adaptive linear unit | Adaptive rectifiers | ≈ [−0.5, ∞) | – | ✓ | – | |
| Adaptive quadratic linear unit | Adaptive self-gated | ≈ [−0.12, ∞) | ✓ | – | – | |
| Adaptive radial basis function | Rational, fractional & other adaptive families | ≈ [0, 1] | ✓ | – | – | |
| Adaptive shifted ReLU | Adaptive rectifiers | ≈ [−0.2, ∞) | – | ✓ | – | |
| Adaptive slope sigmoidal function | Adaptive sigmoidal | ≈ [0, 1] | ✓ | ✓ | – | |
| ALiSA | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Alternated left ReLU | Adaptive rectifiers | ≈ [0, ∞) | – | – | – | |
| Amplifying sine unit | Periodic & radial | (−∞, ∞) | ✓ | – | – | |
| Apical dendrite activation | Rectifier variants | ≈ [0, 1] | – | – | – | |
| APL | Learned & adaptive | learned | – | – | – | |
| Approximate ReLU | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | – | ✓ | – | |
| Aranda-Ordaz | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Arctan | Sigmoidal & saturating | (−π/2, π/2) | ✓ | ✓ | – | |
| Arctan SiLU | Swish & Mish descendants | ≈ [−0.27, 1.57] | ✓ | – | – | |
| Arctid | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| ARiA | Adaptive self-gated | ≈ [−0.07, ∞) | ✓ | – | – | |
| Attention-based ReLU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Bendable linear unit | Adaptive rectifiers | (−∞, ∞) | ✓ | – | – | |
| Bent identity | Other research functions | (−∞, ∞) | ✓ | ✓ | – | |
| Bi-firing | Rectifier variants | ≈ [0, ∞) | ✓ | – | – | |
| Bi-modal sigmoid | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Bilinear | Gated linear units | (−∞, ∞) | ✓ | – | – | |
| Binary step | Linear & threshold | {0, 1} | – | ✓ | – | |
| Bounded bi-firing | Rectifier variants | ≈ [0, 3] | – | – | – | |
| Bounded leaky ReLU | Rectifier variants | (−∞, ∞) | – | ✓ | – | |
| Bounded ReLU | Rectifier variants | ≈ [0, 2] | – | ✓ | – | |
| BReLUPlus | Polynomial, algebraic & logarithmic | ≈ [0, 1] | ✓ | ✓ | – | |
| Catalytic activation | Rational, fractional & other adaptive families | (−∞, ∞) | – | ✓ | – | |
| Cauchy linear unit | Swish & Mish descendants | ≈ [−0.32, ∞) | ✓ | ✓ | – | |
| CELU | Rectifiers (ReLU family) | (−α, ∞) | ✓ | ✓ | nn.CELU | |
| Chebyshev polynomial activation | Rational, fractional & other adaptive families | (−∞, ∞) | ✓ | – | – | |
| Collapsing linear unit | Swish & Mish descendants | ≈ [−0.38, ∞) | ✓ | – | – | |
| Comb-H-sine | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| Complementary log-log | Sigmoidal & saturating | (0, 1) | ✓ | ✓ | – | |
| Cosid | Periodic & radial | (−∞, ∞) | ✓ | ✓ | – | |
| Cosine | Periodic & radial | [−1, 1] | ✓ | – | – | |
| Cosinu-sigmoidal linear unit | Adaptive self-gated | ≈ [−0.29, ∞) | ✓ | – | – | |
| CReLU | Rectifiers (ReLU family) | [0, ∞)² | – | – | – | |
| Cube | Other research functions | (−∞, ∞) | ✓ | ✓ | – | |
| Decaying sine unit | Periodic & radial | ≈ [−1.64, 1.64] | ✓ | – | – | |
| Delay ReLU | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| DELU | Adaptive rectifiers | ≈ [−0.28, ∞) | – | – | – | |
| Derivative of sigmoid | Swish & Mish descendants | ≈ [0, 0.25] | ✓ | – | – | |
| Difference ELU | Rectifier variants | ≈ [−0.14, ∞) | – | – | – | |
| Displaced ReLU | Rectifier variants | ≈ [−0.5, ∞) | – | ✓ | – | |
| DoubleSiLU | Swish & Mish descendants | (−∞, ∞) | ✓ | ✓ | – | |
| dSiLU | Swish & Mish descendants | ≈ [−0.1, 1.1] | ✓ | – | – | |
| Dual Line | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Dual parametric ReLU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| E-swish | Swish & Mish descendants | ≈ [−0.38, ∞) | ✓ | – | – | |
| E-Tanh | Polynomial, algebraic & logarithmic | ≈ [−0.15, ∞) | ✓ | – | – | |
| EANAF | Swish & Mish descendants | ≈ [−0.16, ∞) | ✓ | – | – | |
| EIS-1 | Rational, fractional & other adaptive families | ≈ [−0.22, ∞) | ✓ | – | – | |
| EIS-2 | Rational, fractional & other adaptive families | ≈ [−0.23, ∞) | ✓ | – | – | |
| EIS-3 | Rational, fractional & other adaptive families | ≈ [−0.16, ∞) | ✓ | – | – | |
| ELiSH | Swish & Mish descendants | ≈ [−0.17, ∞) | ✓ | – | – | |
| Elliott function | Sigmoid variants | (0, 1) | ✓ | ✓ | – | |
| ELU | Rectifiers (ReLU family) | (−α, ∞) | ✓ | ✓ | nn.ELU | |
| ELU + parameterised softplus | Adaptive rectifiers | ≈ [−0.75, ∞) | ✓ | ✓ | – | |
| ELUPlus | Polynomial, algebraic & logarithmic | ≈ [0.024, ∞) | ✓ | – | – | |
| Enhanced rational activation | Rational, fractional & other adaptive families | (−∞, ∞) | ✓ | – | – | |
| Erf-based ReLU | Adaptive rectifiers | ≈ [−0.5, ∞) | – | ✓ | – | |
| ErfAct | Adaptive self-gated | ≈ [−0.41, ∞) | ✓ | – | – | |
| Error function | Sigmoidal & saturating | (−1, 1) | ✓ | ✓ | – | |
| Even power-linear (EPLAF) | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | ✓ | – | – | |
| Evolved tanh–ReLU | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | – | – | – | |
| Expcos | Rational, fractional & other adaptive families | ≈ [−0.61, 1] | ✓ | – | – | |
| ExpExpish | Swish & Mish descendants | ≈ [−0.097, ∞) | ✓ | – | – | |
| Exponential | Output & normalising | (0, ∞) | ✓ | ✓ | exp | |
| Exponential distance activation | Rational, fractional & other adaptive families | ≈ [0, 1] | – | – | – | |
| Exponential swish | Swish & Mish descendants | ≈ [0, 1] | ✓ | ✓ | – | |
| Extended ELU | Adaptive rectifiers | ≈ [−0.77, ∞) | – | ✓ | – | |
| Fast ELU | Adaptive rectifiers | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Faster power function linear unit | Rectifier variants | ≈ [−0.3, ∞) | ✓ | – | – | |
| Flatted-T swish | Rectifier variants | ≈ [−0.2, ∞) | ✓ | ✓ | – | |
| Flexible ReLU | Adaptive rectifiers | ≈ [−0.4, ∞) | – | ✓ | – | |
| Fourier series activation | Rational, fractional & other adaptive families | ≈ [−0.72, 1.3] | ✓ | – | – | |
| Fractional ReLU | Rational, fractional & other adaptive families | ≈ [0, ∞) | – | ✓ | – | |
| Gated PReLU–PELU mix | Adaptive rectifiers | ≈ [−1, ∞) | – | ✓ | – | |
| Gated tanh unit | Gated linear units | (−1, 1) | ✓ | – | – | |
| Gating adaptive blending unit | Rational, fractional & other adaptive families | ≈ [−0.27, ∞) | – | ✓ | – | |
| Gaussian | Periodic & radial | (0, 1] | ✓ | – | – | |
| Gaussian error unit | Adaptive sigmoidal | ≈ [0, 1] | ✓ | ✓ | – | |
| Gaussian ReLU | Adaptive rectifiers | (−∞, ∞) | – | – | – | |
| GEGLU | Gated linear units | (−∞, ∞) | ✓ | – | – | |
| GELU | Smooth self-gated | ≈ [−0.17, ∞) | ✓ | – | nn.GELU | |
| GELU (tanh approximation) | Smooth self-gated | ≈ [−0.17, ∞) | ✓ | – | nn.GELU(approximate='tanh') | |
| Generalised hyperbolic tangent | Adaptive sigmoidal | ≈ [−1.5, 1.5] | ✓ | ✓ | – | |
| Generalised ReLU (GReLU) | Adaptive rectifiers | ≈ [0, ∞) | ✓ | ✓ | – | |
| Generalised Riccati activation | Adaptive sigmoidal | ≈ [0, 1] | ✓ | – | – | |
| Generalised swish | Swish & Mish descendants | (−∞, ∞) | ✓ | ✓ | – | |
| Gish | Swish & Mish descendants | ≈ [−0.28, ∞) | ✓ | – | – | |
| GLU | Gated linear units | (−∞, ∞) | ✓ | – | nn.GLU | |
| Growing cosine unit | Periodic & radial | (−∞, ∞) | ✓ | – | – | |
| Gudermannian | Sigmoidal & saturating | (−π/2, π/2) | ✓ | ✓ | – | |
| Hard shrink | Shrinkage & thresholding | {0} ∪ (−∞, −λ) ∪ (λ, ∞) | – | ✓ | nn.Hardshrink | |
| Hard sigmoid | Sigmoidal & saturating | [0, 1] | – | ✓ | nn.Hardsigmoid | |
| Hard swish | Smooth self-gated | ≈ [−0.38, ∞) | – | – | nn.Hardswish | |
| Hard tanh | Sigmoidal & saturating | [−1, 1] | – | ✓ | nn.Hardtanh | |
| Hard-Swish (adaptive) | Adaptive rectifiers | ≈ [−0.62, ∞) | – | – | – | |
| HardELiSH | Swish & Mish descendants | ≈ [−0.1, ∞) | – | – | – | |
| HardshrinkPlus | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | – | – | |
| HardSReLUE | Rectifier variants | ≈ [−0.5, ∞) | – | ✓ | – | |
| HardTanhPlus | Polynomial, algebraic & logarithmic | ≈ [−1, 1] | ✓ | ✓ | – | |
| Hat | Adaptive rectifiers | ≈ [0, 1] | – | – | – | |
| HcLSH | Polynomial, algebraic & logarithmic | ≈ [−0.69, ∞) | ✓ | ✓ | – | |
| Hermite polynomial activation | Rational, fractional & other adaptive families | (−∞, ∞) | ✓ | – | – | |
| Hexpo | Sigmoid variants | ≈ [−1, 1] | ✓ | ✓ | – | |
| Horizontally shifted ELU | Rectifier variants | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Horizontally shifted HardTanh | Rectifier variants | ≈ [−1, 1] | – | ✓ | – | |
| Hyper-sinh | Polynomial, algebraic & logarithmic | (−∞, ∞) | – | ✓ | – | |
| Identity | Linear & threshold | (−∞, ∞) | ✓ | ✓ | nn.Identity | |
| Improved logistic sigmoid | Sigmoid variants | (−∞, ∞) | – | ✓ | – | |
| Inverse multiquadric | Periodic & radial | (0, 1/c] | ✓ | – | – | |
| Inverse polynomial linear unit | Rectifier variants | ≈ [−0.5, ∞) | ✓ | – | – | |
| ISRLU | Rectifiers (ReLU family) | (−1/√α, ∞) | ✓ | ✓ | – | |
| ISRU | Sigmoidal & saturating | (−1/√α, 1/√α) | ✓ | ✓ | – | |
| Kernel activation function | Rational, fractional & other adaptive families | ≈ [−0.045, 3.01] | ✓ | – | – | |
| Laplace linear unit | Swish & Mish descendants | ≈ [−0.18, ∞) | ✓ | – | – | |
| Leaky apical dendrite activation | Rectifier variants | ≈ (−∞, 1] | – | – | – | |
| Leaky learnable ReLU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Leaky parametric SELU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Leaky ReLU | Rectifiers (ReLU family) | (−∞, ∞) | – | ✓ | nn.LeakyReLU | |
| Leaky SELU | Rectifier variants | (−∞, ∞) | – | ✓ | – | |
| Leaky single-peaked triangle linear unit | Rectifier variants | ≈ (−∞, 1] | – | – | – | |
| Learnable extended activation function | Rational, fractional & other adaptive families | ≈ [−0.26, ∞) | ✓ | – | – | |
| LeCun tanh | Sigmoidal & saturating | (−1.7159, 1.7159) | ✓ | ✓ | – | |
| Legendre polynomial activation | Rational, fractional & other adaptive families | (−∞, ∞) | ✓ | – | – | |
| Li-ReLU | Rectifier variants | (−∞, ∞) | – | ✓ | – | |
| LiSHT | Swish & Mish descendants | [0, ∞) | ✓ | – | – | |
| Locally adaptive activation function | Rational, fractional & other adaptive families | depends on g | – | ✓ | – | |
| Log-sigmoid | Sigmoidal & saturating | (−∞, 0) | ✓ | ✓ | nn.LogSigmoid | |
| Log-softmax | Output & normalising | (−∞, 0) | ✓ | ✓ | nn.LogSoftmax | |
| Logarithmic activation (symlog) | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| Logish | Swish & Mish descendants | ≈ [−0.25, ∞) | ✓ | – | – | |
| LogishPlus | Polynomial, algebraic & logarithmic | ≈ [−0.14, ∞) | ✓ | – | – | |
| LogLog | Sigmoid variants | (0, 1) | ✓ | ✓ | – | |
| LogLogish | Swish & Mish descendants | ≈ [−0.31, ∞) | ✓ | – | – | |
| Logmoid activation unit | Adaptive self-gated | ≈ [−0.47, ∞) | ✓ | – | – | |
| LPSELU with reposition | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| LReLUPlus | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| LS-ReLU | Rectifier variants | ≈ [−1, ∞) | – | ✓ | – | |
| Margin ReLU | Adaptive rectifiers | ≈ [−0.5, ∞) | – | ✓ | – | |
| Maxout | Learned & adaptive | learned | – | – | – | |
| Maxsig | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| Mexican ReLU | Adaptive rectifiers | (−∞, ∞) | – | – | – | |
| Minsin | Rectifier variants | ≈ (−∞, 1] | ✓ | – | – | |
| Mirrored ReLU | Rectifier variants | ≈ [0, 1] | – | – | – | |
| Mish | Smooth self-gated | ≈ [−0.31, ∞) | ✓ | – | nn.Mish | |
| MishPlus | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | ✓ | – | – | |
| Mishra activation | Polynomial, algebraic & logarithmic | ≈ [−0.12, 1] | ✓ | – | – | |
| Mixture of Gaussian unit | Rational, fractional & other adaptive families | ≈ [−0.35, 0.57] | ✓ | – | – | |
| Modified arcsinh | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| Modified Elliott function | Sigmoid variants | ≈ [−0.5, 1.5] | ✓ | ✓ | – | |
| Modified leaky ReLU | Rectifier variants | ≈ [−0.49, ∞) | – | – | – | |
| Modified Mexican ReLU | Adaptive rectifiers | ≈ [0, ∞) | – | – | – | |
| Modified SiLU | Swish & Mish descendants | ≈ [−0.26, ∞) | ✓ | – | – | |
| Modified Weibull function | Rational, fractional & other adaptive families | ≈ [0, 0.43] | ✓ | – | – | |
| Multiple parametric ELU | Adaptive rectifiers | ≈ [−1, ∞) | – | ✓ | – | |
| Multiquadratic activation | Rational, fractional & other adaptive families | ≈ [1, ∞) | ✓ | – | – | |
| Multiquadric | Periodic & radial | [c, ∞) | ✓ | – | – | |
| Multistate activation function | Sigmoid variants | ≈ [−1, 1] | ✓ | ✓ | – | |
| N-activation | Adaptive rectifiers | (−∞, ∞) | – | – | – | |
| Natural-logarithm ReLU | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| Negative CReLU | Rectifier variants | ℝ² | – | ✓ | – | |
| Negative exponential | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | ✓ | ✓ | – | |
| Neuron-adaptive activation function | Rational, fractional & other adaptive families | ≈ [0, 1.56] | ✓ | – | – | |
| NewSigmoid | Sigmoid variants | ≈ [−0.71, 0.71] | ✓ | ✓ | – | |
| Non-monotonic cubic unit | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | – | – | |
| Odd power-linear (OPLAF) | Polynomial, algebraic & logarithmic | ≈ [−0.5, ∞) | – | – | – | |
| Optimal activation function | Rectifier variants | ≈ [−0.28, ∞) | – | – | – | |
| P+FELU | Adaptive rectifiers | ≈ [−1.2, ∞) | ✓ | ✓ | – | |
| P-E2-ReLU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| P-SIG-RAMP | Adaptive sigmoidal | ≈ [0, 1] | – | ✓ | – | |
| Padé activation unit | Learned & adaptive | learned | – | – | – | |
| Paired ReLU | Adaptive rectifiers | [0, ∞)² | – | – | – | |
| Pan function | Rectifier variants | ≈ [0, ∞) | – | – | – | |
| PanPlus | Polynomial, algebraic & logarithmic | ≈ [0.049, ∞) | ✓ | – | – | |
| Parameterised self-circulating gating unit | Adaptive self-gated | ≈ [−0.27, ∞) | ✓ | – | – | |
| Parameterised softplus (s+2L) | Adaptive rectifiers | ≈ [−0.5, ∞) | ✓ | ✓ | – | |
| Parametric deformable ELU | Adaptive rectifiers | ≈ [−1, ∞) | ✓ | – | – | |
| Parametric first power linear unit | Adaptive rectifiers | ≈ [−0.02, ∞) | ✓ | – | – | |
| Parametric flatted-T swish | Adaptive rectifiers | ≈ [−0.2, ∞) | ✓ | ✓ | – | |
| Parametric flatten-p Mish | Adaptive rectifiers | ≈ [−0.2, ∞) | ✓ | ✓ | – | |
| Parametric GELU | Adaptive self-gated | ≈ [−0.34, ∞) | ✓ | – | – | |
| Parametric Logish | Swish & Mish descendants | ≈ [−0.025, ∞) | ✓ | – | – | |
| Parametric rectified exponential unit | Adaptive rectifiers | ≈ [−0.88, ∞) | ✓ | – | – | |
| Parametric rectified non-linear unit | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| Parametric scaled tanh | Adaptive self-gated | ≈ [−0.14, ∞) | ✓ | – | – | |
| Parametric SechSig | Sigmoid variants | ≈ [−0.22, ∞) | ✓ | – | – | |
| Parametric SELU | Adaptive rectifiers | ≈ [−1.76, ∞) | – | ✓ | – | |
| Parametric Serf | Adaptive self-gated | ≈ [−0.51, ∞) | ✓ | – | – | |
| Parametric shifted ELU (horizontal) | Adaptive rectifiers | ≈ [−1.5, ∞) | ✓ | ✓ | – | |
| Parametric shifted ELU (vertical) | Adaptive rectifiers | ≈ [−1.8, ∞) | ✓ | ✓ | – | |
| Parametric shifted SiLU | Adaptive self-gated | ≈ [−0.085, ∞) | ✓ | – | – | |
| Parametric sigmoid | Adaptive sigmoidal | ≈ [0, 2] | ✓ | ✓ | – | |
| Parametric sigmoid function | Adaptive sigmoidal | ≈ [0, 1] | ✓ | ✓ | – | |
| Parametric softplus | Rectifier variants | ≈ [−0.69, ∞) | ✓ | ✓ | – | |
| Parametric swish (p-swish) | Adaptive rectifiers | ≈ [−0.28, ∞) | – | – | – | |
| Parametric tanh linear unit | Adaptive rectifiers | ≈ [−0.5, ∞) | ✓ | ✓ | – | |
| Parametric TanhSig | Sigmoid variants | ≈ [−0.35, ∞) | ✓ | – | – | |
| Parametric variational linear unit | Adaptive rectifiers | ≈ [−0.1, ∞) | – | – | – | |
| PATS | Adaptive self-gated | ≈ [−0.42, ∞) | ✓ | – | – | |
| PELU | Learned & adaptive | (−a, ∞) | ✓ | ✓ | – | |
| Penalised tanh | Other research functions | (−a, 1) | – | ✓ | – | |
| Phish | Swish & Mish descendants | ≈ [0, ∞) | ✓ | – | – | |
| Piecewise linear unit | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Piecewise linear unit (PLU) | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Piecewise Mexican-hat | Periodic & radial | ≈ [−0.39, 0.87] | ✓ | – | – | |
| Piecewise radial basis function | Periodic & radial | ≈ [0, 1] | ✓ | – | – | |
| Polyexp | Polynomial, algebraic & logarithmic | ≈ [−0.22, 1.29] | ✓ | – | – | |
| Polynomial linear unit | Rectifier variants | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Polynomial universal activation | Polynomial, algebraic & logarithmic | ≈ [−0.25, ∞) | – | – | – | |
| Positive PReLU | Adaptive rectifiers | ≈ [0, ∞) | – | ✓ | – | |
| Power function linear unit | Rectifier variants | ≈ [−0.15, ∞) | ✓ | – | – | |
| Power linear unit | Rectifier variants | ≈ [−1, ∞) | ✓ | ✓ | – | |
| PReLU | Learned & adaptive | (−∞, ∞) | – | ✓ | nn.PReLU | |
| Probit (normal CDF) | Sigmoidal & saturating | (0, 1) | ✓ | ✓ | – | |
| PTaLU | Adaptive rectifiers | ≈ [−0.64, ∞) | – | ✓ | – | |
| QuickGELU | Smooth self-gated | ≈ [−0.16, ∞) | ✓ | – | – | |
| React-PReLU | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| Rectified BLU | Adaptive rectifiers | ≈ [0, ∞) | – | ✓ | – | |
| Rectified exponential unit | Rectifier variants | ≈ [−0.37, ∞) | ✓ | – | – | |
| Rectified hyperbolic secant | Swish & Mish descendants | ≈ [−0.66, 0.66] | ✓ | – | – | |
| Rectified linear tanh | Adaptive rectifiers | (−∞, ∞) | ✓ | ✓ | – | |
| Rectified parametric sigmoid unit | Adaptive rectifiers | ≈ [0, ∞) | – | ✓ | – | |
| Rectified power unit | Polynomial, algebraic & logarithmic | [0, ∞) | – | ✓ | – | |
| Rectified softplus | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| ReGLU | Gated linear units | (−∞, ∞) | – | – | – | |
| ReLU | Rectifiers (ReLU family) | [0, ∞) | – | ✓ | nn.ReLU | |
| ReLU memristor-like activation | Adaptive rectifiers | ≈ [−0.72, ∞) | ✓ | – | – | |
| ReLU6 | Rectifiers (ReLU family) | [0, 6] | – | ✓ | nn.ReLU6 | |
| Ricker wavelet | Periodic & radial | [−2e−3/2, 1] ≈ [−0.45, 1] | ✓ | – | – | |
| Root2sigmoid | Sigmoid variants | ≈ [−0.25, 0.25] | ✓ | ✓ | – | |
| Rootsig | Sigmoid variants | ≈ [−1, 1] | ✓ | ✓ | – | |
| Rowdy activation | Rational, fractional & other adaptive families | ≈ [−0.26, ∞) | – | – | – | |
| RReLU | Rectifiers (ReLU family) | (−∞, ∞) | – | ✓ | nn.RReLU | |
| RSigELU | Rectifier variants | ≈ [−0.5, ∞) | – | ✓ | – | |
| RSigELUD | Rectifier variants | ≈ [−0.5, ∞) | – | ✓ | – | |
| RSign | Adaptive sigmoidal | {−1, 1} | – | ✓ | – | |
| Saha-Bora activation | Polynomial, algebraic & logarithmic | ≈ [0.67, 1] | ✓ | – | – | |
| SaRa | Rectifier variants | ≈ [−0.66, ∞) | – | – | – | |
| Scaled and shifted sine | Periodic & radial | ≈ [−0.2, 0.8] | ✓ | – | – | |
| Scaled exponential function | Rational, fractional & other adaptive families | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Scaled exponentially-regularised linear unit | Rectifier variants | ≈ [−0.74, ∞) | – | – | – | |
| Scaled scaled ELU | Rectifier variants | ≈ [−1.76, ∞) | – | ✓ | – | |
| Scaled sine-hyperbolic | Rational, fractional & other adaptive families | (−∞, ∞) | ✓ | ✓ | – | |
| Scaled softsign | Adaptive sigmoidal | ≈ [−2, 2] | ✓ | ✓ | – | |
| Scaled square-root activation | Polynomial, algebraic & logarithmic | (−∞, ∞) | – | ✓ | – | |
| Scaled-gamma-tanh | Adaptive rectifiers | (−∞, ∞) | – | ✓ | – | |
| SechSig | Sigmoid variants | ≈ [−0.21, ∞) | ✓ | – | – | |
| Self arctan | Swish & Mish descendants | ≈ [0, ∞) | ✓ | – | – | |
| Self-learnable activation (polynomial) | Rational, fractional & other adaptive families | ≈ [−0.11, ∞) | ✓ | – | – | |
| SELU | Rectifiers (ReLU family) | (−λα, ∞) ≈ (−1.76, ∞) | – | ✓ | nn.SELU | |
| Serf | Swish & Mish descendants | ≈ [−0.35, ∞) | ✓ | – | – | |
| Shape-autotuning adaptive activation | Adaptive sigmoidal | ≈ [−0.97, 3] | ✓ | – | – | |
| Shape-autotuning sigmoid | Adaptive sigmoidal | ≈ [−1, 1] | ✓ | ✓ | – | |
| Shifted and scaled sigmoid | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Shifted quadratic unit | Polynomial, algebraic & logarithmic | ≈ [−0.25, ∞) | ✓ | – | – | |
| Shifted ReLU | Rectifier variants | ≈ [−1, ∞) | – | ✓ | – | |
| Shifted sinc unit | Periodic & radial | ≈ [−0.68, 3.14] | ✓ | – | – | |
| Shifted softplus | Rectifiers (ReLU family) | (−ln 2, ∞) | ✓ | ✓ | – | |
| SiELU | Swish & Mish descendants | ≈ [−0.17, ∞) | ✓ | – | – | |
| SigLin | Sigmoid variants | (−∞, ∞) | ✓ | ✓ | – | |
| SigLU | Rectifier variants | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Sigmoid | Sigmoidal & saturating | (0, 1) | ✓ | ✓ | nn.Sigmoid | |
| Sigmoid-Algebraic | Sigmoid variants | ≈ [0.27, 0.73] | ✓ | ✓ | – | |
| Sigmoid-Gumbel | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Sigmoid–tanh combinations | Sigmoid variants | depends on the pair | – | ✓ | – | |
| Sign | Linear & threshold | {−1, 0, 1} | – | ✓ | – | |
| SignReLU | Rectifier variants | ≈ [−1, ∞) | ✓ | ✓ | – | |
| SignReLUPlus | Polynomial, algebraic & logarithmic | ≈ [−1, ∞) | ✓ | ✓ | – | |
| SiLU / Swish | Smooth self-gated | ≈ [−0.28, ∞) | ✓ | – | nn.SiLU | |
| Sinc | Periodic & radial | ≈ [−0.22, 1] | ✓ | – | – | |
| Sinc-Sigmoid | Sigmoid variants | ≈ [0.84, 1] | ✓ | ✓ | – | |
| Sincos | Rational, fractional & other adaptive families | ≈ [−1.5, 0.75] | ✓ | – | – | |
| Sine (SIREN) | Periodic & radial | [−1, 1] | ✓ | – | – | |
| Sine + sigmoid (CSS) | Rational, fractional & other adaptive families | ≈ [−0.5, 1.5] | ✓ | – | – | |
| SineReLU | Rectifier variants | ≈ [−0.71, ∞) | – | – | – | |
| Sine–Gaussian–sigmoid combination | Rational, fractional & other adaptive families | ≈ [−0.5, 1.6] | ✓ | – | – | |
| SinLU | Periodic & radial | ≈ [−0.5, ∞) | ✓ | – | – | |
| SinP | Periodic & radial | (−∞, ∞) | ✓ | ✓ | – | |
| SinSig | Swish & Mish descendants | ≈ [−0.43, ∞) | ✓ | – | – | |
| Slope varying activation function | Adaptive sigmoidal | ≈ [−1, 1] | ✓ | ✓ | – | |
| Sloped ReLU | Rectifier variants | ≈ [0, ∞) | – | ✓ | – | |
| SmeLU | Smooth self-gated | [0, ∞) | ✓ | ✓ | – | |
| Smish | Swish & Mish descendants | ≈ [−0.25, ∞) | ✓ | – | – | |
| Smooth activation unit | Adaptive rectifiers | (−∞, ∞) | ✓ | ✓ | – | |
| Smooth maximum unit | Adaptive rectifiers | (−∞, ∞) | ✓ | ✓ | – | |
| Smooth step | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Snake | Periodic & radial | (−∞, ∞) | ✓ | ✓ | – | |
| Soft clipping | Sigmoid variants | ≈ [0, 1] | ✓ | ✓ | – | |
| Soft clipping Mish | Adaptive rectifiers | ≈ [0, ∞) | – | ✓ | – | |
| Soft clipping Swish | Adaptive rectifiers | ≈ [0, ∞) | – | ✓ | – | |
| Soft exponential | Learned & adaptive | depends on α | ✓ | ✓ | – | |
| Soft shrink | Shrinkage & thresholding | (−∞, ∞) | – | ✓ | nn.Softshrink | |
| Soft++ | Rectifier variants | (−∞, ∞) | ✓ | ✓ | – | |
| Soft-root-sign | Sigmoid variants | ≈ [−2.46, 2] | ✓ | – | – | |
| Softmax | Output & normalising | (0, 1), sums to 1 | ✓ | ✓ | nn.Softmax | |
| Softmin | Output & normalising | (0, 1), sums to 1 | ✓ | ✓ | nn.Softmin | |
| SoftModulusQ | Rectifier variants | ≈ [0, ∞) | ✓ | – | – | |
| SoftModulusT | Rectifier variants | ≈ [0, ∞) | ✓ | – | – | |
| Softplus | Rectifiers (ReLU family) | (0, ∞) | ✓ | ✓ | nn.Softplus | |
| Softplus linear unit | Rectifier variants | ≈ [−1.39, ∞) | ✓ | ✓ | – | |
| SoftshrinkPlus | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| Softsign | Sigmoidal & saturating | (−1, 1) | ✓ | ✓ | nn.Softsign | |
| SoftsignPlus | Polynomial, algebraic & logarithmic | ≈ [−1, 1] | ✓ | ✓ | – | |
| Sparsemax | Output & normalising | [0, 1], sums to 1 | – | ✓ | – | |
| SPLASH | Adaptive rectifiers | ≈ [0, ∞) | – | – | – | |
| SPOCU | Polynomial, algebraic & logarithmic | ≈ (−∞, 0.76] | – | – | – | |
| SQNL | Other research functions | [−1, 1] | ✓ | ✓ | – | |
| Square linear unit | Polynomial, algebraic & logarithmic | ≈ [−1, ∞) | ✓ | ✓ | – | |
| Square logistic sigmoid | Polynomial, algebraic & logarithmic | [0, 1] | ✓ | ✓ | – | |
| Square REU | Polynomial, algebraic & logarithmic | ≈ [−0.5, ∞) | – | – | – | |
| Square softmax | Output & normalising | ≈ [0, 1] | ✓ | – | – | |
| Square softplus | Polynomial, algebraic & logarithmic | ≈ [0, ∞) | ✓ | ✓ | – | |
| Square-root activation | Polynomial, algebraic & logarithmic | (−∞, ∞) | – | ✓ | – | |
| Squared ReLU | Smooth self-gated | [0, ∞) | ✓ | ✓ | – | |
| Squareplus | Smooth self-gated | (0, ∞) | ✓ | ✓ | – | |
| Squish | Polynomial, algebraic & logarithmic | ≈ [−0.5, ∞) | – | – | – | |
| SReLU | Learned & adaptive | (−∞, ∞) | – | ✓ | – | |
| SReLUPlus | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| STAC-tanh | Adaptive sigmoidal | (−∞, ∞) | – | ✓ | – | |
| StarReLU | Smooth self-gated | [b, ∞) | ✓ | ✓ | – | |
| StepPlus | Polynomial, algebraic & logarithmic | ≈ [0, 1] | ✓ | ✓ | – | |
| Suish | Swish & Mish descendants | ≈ [−0.37, ∞) | ✓ | – | – | |
| SwAT | Swish & Mish descendants | (−∞, ∞) | ✓ | ✓ | – | |
| SwiGLU | Gated linear units | (−∞, ∞) | ✓ | – | – | |
| Swim | Adaptive self-gated | (−∞, ∞) | ✓ | ✓ | – | |
| Swish-β | Learned & adaptive | (−∞, ∞) | ✓ | – | – | |
| SwishPlus | Polynomial, algebraic & logarithmic | ≈ [−0.15, ∞) | ✓ | – | – | |
| Symexp | Polynomial, algebraic & logarithmic | (−∞, ∞) | ✓ | ✓ | – | |
| Symmetrical GELU | Swish & Mish descendants | ≈ [0, ∞) | ✓ | – | – | |
| Tangent linear unit | Adaptive rectifiers | ≈ [−0.91, ∞) | – | ✓ | – | |
| Tangent-bipolar-sigmoid ReLU | Swish & Mish descendants | ≈ [0, ∞) | ✓ | – | – | |
| Tangent-sigmoid ReLU | Swish & Mish descendants | ≈ [−0.27, ∞) | ✓ | – | – | |
| Tanh | Sigmoidal & saturating | (−1, 1) | ✓ | ✓ | nn.Tanh | |
| Tanh linear unit | Rectifier variants | ≈ [−1, ∞) | – | ✓ | – | |
| Tanh shrink | Shrinkage & thresholding | (−∞, ∞) | ✓ | ✓ | nn.Tanhshrink | |
| Tanh SiLU | Swish & Mish descendants | ≈ [−0.27, 1] | ✓ | – | – | |
| Tanh-based ReLU | Adaptive rectifiers | ≈ [−1, ∞) | – | ✓ | – | |
| TanhExp | Swish & Mish descendants | ≈ [−0.35, ∞) | ✓ | – | – | |
| TanhLU | Adaptive rectifiers | (−∞, ∞) | ✓ | ✓ | – | |
| TanhSig | Sigmoid variants | ≈ [−0.47, ∞) | ✓ | – | – | |
| TanhSoft-1 | Adaptive self-gated | ≈ [−0.14, ∞) | ✓ | – | – | |
| TanhSoft-2 | Adaptive self-gated | ≈ [−0.18, ∞) | ✓ | – | – | |
| TanhSoft-3 | Adaptive self-gated | ≈ [−0.36, ∞) | ✓ | – | – | |
| TBSReLU (learnable) | Adaptive self-gated | ≈ [0, ∞) | ✓ | – | – | |
| TeLU | Adaptive rectifiers | ≈ [0, ∞) | ✓ | – | – | |
| Tent | Adaptive rectifiers | ≈ [0, 1] | – | – | – | |
| Thresholded ReLU | Rectifiers (ReLU family) | {0} ∪ (θ, ∞) | – | ✓ | nn.Threshold | |
| Trainable amplitude | Rational, fractional & other adaptive families | ≈ [−1.7, 2.3] | ✓ | ✓ | – | |
| Trainable compound activation | Rational, fractional & other adaptive families | ≈ [−0.5, ∞) | – | ✓ | – | |
| Transformative adaptive activation function | Rational, fractional & other adaptive families | ≈ [−1.3, 1.7] | ✓ | ✓ | – | |
| Triple-state sigmoid | Sigmoid variants | ≈ [0, 3] | ✓ | ✓ | – | |
| Triple-state swish | Swish & Mish descendants | ≈ [−0.094, ∞) | ✓ | – | – | |
| Truncated Gaussian unit | Rational, fractional & other adaptive families | depends on ξ | ✓ | ✓ | – | |
| Tunable swish | Adaptive rectifiers | ≈ [−0.42, ∞) | – | – | – | |
| Universal activation function | Rational, fractional & other adaptive families | ≈ [0, 1] | ✓ | ✓ | – | |
| Variant sigmoid function | Sigmoid variants | ≈ [−1, 1] | ✓ | ✓ | – | |
| Variational linear unit | Rectifier variants | ≈ [−0.25, ∞) | – | – | – | |
| Vertically shifted ELU | Rectifier variants | ≈ [−1.3, ∞) | ✓ | ✓ | – | |
| Vertically shifted HardTanh | Rectifier variants | ≈ [−0.5, 1.5] | – | ✓ | – | |
| Wave | Periodic & radial | ≈ [−0.14, 1] | ✓ | – | – |
Most functions come from the survey by Kunc & Kléma (2024), which collects about 400 activation functions. These 52 entries from it have no card because they can't be drawn as one fixed curve of one input:
| Function | Why it isn't plotted |
|---|---|
| β-softmax | vector function with a randomly sampled exponent |
| Softsign randomised leaky ReLU (S-RReLU) | as written it has a pole at x = −1 and a randomly sampled slope |
| Noisy ReLU (NReLU) | stochastic: adds Gaussian noise scaled to the input |
| Spatial context-aware activation (SCAA) | uses a depth-wise convolution over the feature map |
| Randomly translational ReLU (RT-ReLU) | stochastic shift during training; plain ReLU at test time |
| DualReLU | takes two inputs |
| Orthogonal permutation linear unit (OPLU) | acts on pairs of neurons |
| Elastic ReLU (EReLU) | stochastic slope during training |
| Average biased ReLU (ABReLU) | threshold computed from the data |
| DualELU | takes two inputs |
| Elastic adaptively parametric compounded unit (EACU) | stochastic slope plus trainable parameters |
| Lipschitz ReLU (L-ReLU) | slope chosen from a data-dependent Lipschitz constant |
| Linear quadratic activation (LinQ) | the survey's printed pieces do not join, and the intended form is unclear |
| Rand softplus (RSP) | mixing weight set by the input noise level |
| Knowledge discovery activation (KDAC) | multi-part adaptive definition with clipped interpolation |
| k-winners-take-all (k-WTA) | vector function: keeps the k largest entries |
| Volatility-based activation (VBAF) | takes a window of inputs |
| Hybrid chaotic activation (HCAF) | recursive, chaotic definition |
| Fusion of chaotic activation (FCAF) | recursive, chaotic definition |
| Cascade chaotic activation (CCAF) | recursive, chaotic definition |
| Funnel PReLU (FunPReLU) | uses a spatial (3×3) condition |
| Randomly translational PReLU (RT-PReLU) | stochastic shift during training |
| Probabilistic activation (ProbAct) | stochastic: adds sampled noise |
| Adaptive offset activation (AOAF) | offset computed from the mean input |
| Dynamic leaky ReLU (DLReLU) | slope set from the previous epoch's error |
| Dynamic ReLU (DReLU) | threshold computed per batch |
| Dual parametric family (DPAF) | a recipe for transforming any activation |
| Fully parameterised activation (FPAF) | a recipe combining any two activations |
| Elastic PReLU (EPReLU) | stochastic slope during training |
| Elastic ELU (EELU) | stochastic slope during training |
| Tuned softmax (tsoftmax) | vector function |
| Generalised Lehmer softmax (glsoftmax) | vector function with data-dependent normalisation |
| Generalised power softmax (gpsoftmax) | vector function with data-dependent normalisation |
| Noisy activation functions | stochastic noise injected in the saturated regime |
| Fractional softplus, tanh, SiLU, GELU and ALU | defined by fractional derivatives without a closed form |
| Fractional leaky / parametric ReLU and ELU | complex-valued for negative inputs as written |
| Global-local neuron (GLN) | the local component is not fully specified |
| Multi-bias activation (MBA) | vector output with one channel per bias |
| Trapezoid-shaped activation (TSAF) and TSAFPlus | the printed signs do not give the described trapezoid |
| Multi-bin trainable linear unit (MTLU) | a general multi-segment recipe |
| Continuous piecewise nonlinear (CPN) | a general multi-segment recipe |
| Look-up table unit (LuTU) | a learned table of points |
| Average of a pool of activations (APAF) | normalised using minimum and maximum over the data |
| Deep Kronecker networks (DKNN) | a framework (its rowdy special case is shown) |
| Randomised Padé activation unit (RPAU) | stochastic coefficients during training |
| Orthogonal Padé activation unit (OPAU) | depends on the chosen orthogonal basis |
| Spline interpolating activations (SAF) | a learned spline through knots |
| MeLUPlus | the survey gives only one smoothed component |
| Variable activation function (VAF) | a small subnetwork |
| Flexible activation bag (FAB) | a subnetwork chooses and parameterises activations |
| Dynamic parameter ReLU (DY-ReLU) | parameters predicted from the input by a hyperfunction |
| SAVE-inspired activations | a list of variations without separate definitions |
The solid line is the function and the dashed line is its derivative, computed numerically from the same formula. The x-range is printed in the corner; the y-axis is scaled to fit and clipped at ±6. Where a function has parameters, the card plots the stated defaults or several values (see the legend). Vector functions such as softmax are plotted for two classes, and gated units with both inputs tied to x.
Smooth means differentiable everywhere (no kinks). Monotonic means never decreasing (or never increasing). Bounded means the output stays in a finite interval in both directions. A torch.nn tag means PyTorch ships it as a module. Chosen for links to the use cases where it's a typical pick.
Each card names the paper usually credited with the function. Cards marked "via Kunc & Kléma, 2024" were collected from the survey Three Decades of Activations: every one of their implementations was checked numerically against the survey's own formula. Where the survey's formula has an evident typo, the card says how it was corrected. Descriptions are our own.
The Nonlinearity Index is a growing catalogue of activation functions for deep learning: the formula, how each curve behaves, and a plot of it and its derivative, organised by family and paired with a guide to which functions get chosen for which tasks. It is built as part of the open ActivationFWorks project and grows in batches toward ≈400 functions.
Corrections to formulas, citations or use-case claims are welcome: open an issue or pull request on GitHub.