No activation function belongs to a domain. What carries over between tasks are properties: smooth gradients, sparsity, bounded outputs, gating, usable higher derivatives. This guide shows which functions practitioners tend to reach for in 19 kinds of tasks and architectures, and explains which property makes each one a good fit.
The feed-forward blocks hold most of a Transformer's parameters, so their activation matters. Smooth, self-gated units such as GELU beat ReLU slightly, and multiplicative GLU gates (SwiGLU, GEGLU) lower perplexity further at equal compute: the gate lets each hidden unit scale its value path by an input-dependent amount. Gating helps most, though Squared ReLU, which has no gate, is also competitive, so the gain isn't only about gating.
Watch out: GLU variants use three weight matrices instead of two; shrink the hidden size to about 2/3 to keep the parameter count equal.
Attention needs non-negative weights that sum to one over the keys. Softmax gives that smoothly but never assigns exact zeros, so some weight leaks to every token. Sparsemax and entmax can zero out irrelevant tokens, which makes attention easier to interpret and can help when only a few positions matter.
Watch out: Always subtract the maximum score before exponentiating; scaling scores by 1/√d keeps softmax out of its saturated regime.
ReLU has been the default since AlexNet: it's cheap, sparse and pairs well with batch normalisation. In larger and deeper models, smooth self-gated activations such as SiLU and Mish usually add a fraction of a percent of accuracy, because their gradients don't switch off abruptly and their small negative lobe keeps some information from negative inputs.
Watch out: Gains from swapping activations are small next to architecture, data augmentation and training schedule changes.
Encoders and decoders mostly use ReLU-family units: sparse activations act like a learned feature selector and stay cheap on large 3D volumes. Medical datasets are small and training often uses tiny batches with instance normalisation, which makes dead units more costly, so a small negative slope (Leaky ReLU, PReLU) is common. Per-voxel sigmoid or softmax outputs give the probabilities that overlap losses such as Dice work with.
Watch out: The output activation must match the label structure: sigmoid for overlapping structures, softmax for mutually exclusive classes.
On phones and microcontrollers, exponentials are slow and activations are often stored as 8-bit integers. Piecewise-linear functions need only comparisons and multiplications, and a bounded range (ReLU6 caps at 6) lets a fixed-point format cover every value without overflow. Hard swish and hard sigmoid keep most of the accuracy of their smooth versions at a fraction of the cost.
Watch out: Check which activations your target runtime fuses into the preceding convolution; an unfused activation can cost more than its arithmetic suggests.
MobileNetV2 (ReLU6) · MobileNetV3 (hard swish, hard sigmoid)
Financial series have a low signal-to-noise ratio, shift between regimes and offer relatively little independent data. Smooth activations give smoother fitted functions and steadier gradients, so the model is less prone to the piecewise artefacts and abrupt switching of ReLU units on noisy inputs. Zero-centred, non-dying units (tanh, ELU, GELU) keep small signals alive, and tanh keeps recurrent state bounded over long sequences. Outputs follow the target: linear for returns, softplus or exponential for strictly positive volatility.
Watch out: Activation choice is a minor lever compared with leakage-free validation, regime shifts and feature design. Don't over-tune it on one backtest.
Gates need values in (0, 1) to act as soft switches that decide how much to keep, write and read, which is exactly the sigmoid's range. The candidate state uses tanh because it's bounded and zero-centred, which stops the cell state from drifting or blowing up as it's updated over hundreds of steps.
Watch out: Saturated gates stop gradients; initialising the forget-gate bias to about 1 helps the cell remember early in training.
The discriminator must pass useful gradients to the generator even for samples it confidently rejects, so a leaky slope (usually 0.2) is standard. The generator's output uses tanh because training images are scaled to [−1, 1], and a bounded output can't produce out-of-range pixels.
Watch out: GAN stability depends far more on the loss, normalisation and learning rates than on activation choice.
The denoiser has to represent a smooth function of the noisy input across many noise levels. SiLU paired with group normalisation became the standard in diffusion U-Nets after DDPM, and diffusion Transformers inherit the GELU MLPs of vision Transformers. Smooth activations avoid the blocky artefacts that hard kinks can introduce into generated images.
Watch out: Keep the activation consistent with any pretrained weights you fine-tune.
DDPM, Stable Diffusion U-Nets (SiLU) · DiT (GELU, tanh approximation)
A plain ReLU MLP fed raw coordinates is biased toward low frequencies, so it blurs fine detail in images, shapes and scenes. Either encode the inputs with Fourier features (NeRF) or use periodic or Gaussian activations directly (SIREN). Sine activations also have smooth, non-zero derivatives of every order, which lets the network fit gradients and Laplacians of signals.
Watch out: Sine networks need their special initialisation and frequency scale ω₀; with a default initialisation they train poorly.
The loss contains the PDE residual, which needs second or higher derivatives of the network's output with respect to its inputs. ReLU's second derivative is zero almost everywhere, so it can't represent terms such as a Laplacian. Infinitely differentiable activations like tanh and sine keep those derivatives informative.
Watch out: Tanh saturates in deep PINNs; adaptive (learned-slope) activations and input scaling are common fixes.
Forces are the negative gradient of the predicted energy, and molecular dynamics integrates them over millions of steps. Smooth activations give smooth potential-energy surfaces and continuous forces; a kink in the activation becomes a discontinuous force, which breaks energy conservation in simulation.
Watch out: Also check that the network's other components (cutoffs, radial bases) are smooth.
Message passing is applied at every node in every layer, so cheap activations are the norm. Graph attention computes its edge scores with a Leaky ReLU (slope 0.2) before the softmax so that negative scores still receive gradient, and uses ELU in hidden layers.
Watch out: Depth, over-smoothing and normalisation usually matter more than the activation in GNNs.
Tabular data rarely suits convolutions, so models are deep fully connected networks. SELU lets those networks train without batch normalisation by keeping activations self-normalised, which helps with small batches and heterogeneous features. Tabular Transformers borrow GLU-style feed-forward blocks.
Watch out: Gradient-boosted trees remain a very strong baseline on tabular data; compare against one.
On-policy methods such as PPO are sensitive to sudden changes in the policy, and many implementations use tanh hidden units because bounded, smooth activations keep updates steady. Off-policy methods (SAC, TD3) typically use ReLU. For continuous actions, a final tanh squashes the output into the action bounds, and discrete policies use a softmax over actions.
Watch out: Implementation details (observation normalisation, initialisation) often outweigh the activation; keep them fixed when comparing.
Stable-Baselines3 PPO (tanh by default) · SAC (ReLU)
Waveforms are dominated by periodic components. Snake adds a periodic term to the identity, which gives the generator an inductive bias toward pitch and harmonics and improves out-of-distribution audio. WaveNet's gated tanh unit lets each layer decide how much of its filtered signal to pass on.
Watch out: Periodic activations need sensible frequency initialisation; with learnable frequencies, monitor them during training.
Iterative sparse-coding algorithms alternate a linear step with soft thresholding, the proximal operator of the L1 penalty. Unrolling those iterations into layers turns the threshold into the activation, so the network inherits the algorithm's sparsity and converges in far fewer steps.
Watch out: The threshold λ is usually learned per layer; initialise it from the regularisation strength.
The output activation encodes the range of the target and pairs with a likelihood: sigmoid with binary cross-entropy, softmax with categorical cross-entropy, identity with squared error, exponential or softplus with Poisson or Gaussian scale parameters. In medical survival analysis, the complementary log-log link gives a discrete-time proportional-hazards model.
Watch out: Use the fused, numerically stable loss (BCE with logits, cross-entropy on raw logits) rather than applying the activation yourself.
With ±1 activations and weights, multiplications become XNOR and bit-count operations, which saves a great deal of energy and memory. Spiking neurons emit binary events, so activity is sparse in time. Both are trained with surrogate gradients: the forward pass uses the hard step, and the backward pass pretends it was a smooth or clipped function such as hard tanh.
Watch out: Expect an accuracy gap compared with full-precision networks, especially on large tasks.
No use case matches those filters.
Property matrix
Which properties each use case relies on. Read down a column to see where a property matters; hover a heading for its definition.
The same information turned around: every function that appears above, with the use cases it's picked for. Most show up in several unrelated areas, which is the point.