Activation Functions: ReLU, Sigmoid, and Beyond

Every neuron in a neural network faces the same question: fire or stay quiet? The activation function answers it. Pick the wrong one and your gradients vanish, your neurons die, or your model trains at half speed. Pick the right one and training converges faster, representations become richer, and generalization improves. This guide covers every activation function that matters in 2026, from the classics that built the field to the modern defaults powering today's largest language models.

We'll track one running example throughout: a feedforward network classifying handwritten digits (MNIST-style, 10 classes). Watching the same architecture respond to different activations makes the performance gaps concrete.

Sigmoid and Tanh Started It All

The sigmoid function was the original neural network activation, borrowed from logistic regression. It squashes any input into the range (0, 1), producing something that looks like a probability.

$\sigma(x) = \frac{1}{1 + e^{-x}}$

Where:

• $\sigma(x)$ is the output of the sigmoid function
• $x$ is the input (the weighted sum of inputs plus bias for a given neuron)
• $e$ is Euler's number (approximately 2.718)

The derivative peaks at $x = 0$ with a maximum of 0.25, then drops off rapidly. This creates the infamous vanishing gradient problem: during backpropagation, gradients get multiplied by this derivative at every layer. Stack 10 layers deep and your gradient shrinks by $0.25^{10} \approx 10^{-6}$. Early layers barely learn.

Sigmoid also has a subtler flaw: its outputs are always positive, so all gradients flowing into a weight matrix carry the same sign, forcing weight updates into a zigzag pattern that slows convergence.

Tanh Fixes the Centering Problem

Tanh addressed sigmoid's centering issue by stretching the output range to (-1, 1).

$\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$

Where:

• $\tanh(x)$ is the output, ranging from -1 to 1
• $x$ is the neuron's pre-activation value
• $e^x$ and $e^{-x}$ are exponential terms that create the symmetric S-curve

Tanh's derivative reaches 1.0 at $x = 0$, four times sigmoid's peak. Gradients survive longer and training is noticeably faster. Still, the vanishing gradient problem persists because the derivative decays toward zero for large inputs.

ReLU Changed Deep Learning Forever

The Rectified Linear Unit is almost embarrassingly simple, yet it made deep networks trainable. Proposed by Nair and Hinton in 2010 and popularized in AlexNet (2012), ReLU remains the most widely used activation in convolutional neural networks.

$\text{ReLU}(x) = \max(0, x)$

Where:

• $x$ is the neuron's pre-activation value
• The output equals $x$ when $x > 0$, and equals $0$when$x \leq 0$

ReLU's gradient is either 1 (positive inputs) or 0 (negative inputs). That constant gradient of 1 is the key: gradients flow backward without shrinking, no matter how many layers they traverse. Training a 50-layer neural network becomes feasible.

ReLU is also computationally cheap: no exponentials, no divisions, just a comparison with zero. This translates to real speedups on GPUs where matrix operations dominate training time.

The Dying ReLU Problem

There's a catch. When a neuron's weighted sum consistently falls below zero, ReLU outputs zero with zero gradient. The neuron stops contributing and stops updating. It's permanently dead. This happens more often than you'd expect: a large negative bias learned early in training, an aggressive learning rate, or poor initialization can kill entire layers. Research has shown that up to 40% of neurons can die in poorly configured networks.

Click to expandDecision tree for choosing the right activation function based on architecture and task

ReLU Variants That Fix the Dying Neuron Problem

Several ReLU modifications address the dying neuron issue while preserving computational efficiency.

Leaky ReLU

Leaky ReLU adds a small slope for negative inputs instead of clamping to zero.

$\text{LeakyReLU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \leq 0 \end{cases}$

Where:

• $x$ is the pre-activation value
• $\alpha$ is a small constant, typically 0.01
• The negative slope ensures a non-zero gradient everywhere

Parametric ReLU (PReLU)

PReLU makes $\alpha$ a learnable parameter. The network decides the optimal negative slope during training. He et al. (2015) showed PReLU improved ImageNet classification by 1.1% over ReLU.

Exponential Linear Unit (ELU)

ELU uses an exponential curve for negative inputs that smoothly approaches $-\alpha$.

$\text{ELU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha(e^x - 1) & \text{if } x \leq 0 \end{cases}$

Where:

• $x$ is the pre-activation value
• $\alpha$ controls the saturation value for negative inputs (default: 1.0)
• $e^x - 1$ creates a smooth exponential curve that approaches $-\alpha$

ELU is smooth everywhere, including at $x = 0$, which helps optimizers find better minima. The exponential computation makes it slightly slower than ReLU per forward pass.

GELU Became the Default for Transformers

The Gaussian Error Linear Unit, introduced by Hendrycks and Gimpel (2016), didn't gain traction until BERT adopted it in 2018. Since then, GELU has become the standard in transformer architectures: GPT-2, GPT-3, RoBERTa, Vision Transformers (ViT), and nearly every large language model built since.

$\text{GELU}(x) = x \cdot \Phi(x) = x \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$

Where:

• $x$ is the input value
• $\Phi(x)$ is the cumulative distribution function of the standard normal distribution
• $\text{erf}$ is the Gaussian error function
• The output smoothly blends the input with a probabilistic gate

The key difference from ReLU: GELU is smooth and non-monotonic. It dips to a minimum of about $-0.17$ near $x \approx -0.75$ before rising back toward zero. This means GELU can output small negative values, which helps the network maintain richer gradient signals during training.

Why did transformers adopt GELU over ReLU? The attention mechanism produces pre-activation distributions that are roughly Gaussian. GELU's probabilistic gating aligns naturally with this distribution, producing smoother optimization surfaces. Empirical studies confirm GELU consistently outperforms ReLU in transformer-based models, often by 0.5-1.5% on downstream benchmarks.

Click to expandComparison of activation function properties across sigmoid, tanh, ReLU, GELU, and SiLU

SiLU and SwiGLU Power Modern Vision and Language Models

SiLU (Swish)

The Sigmoid Linear Unit, also known as Swish, was discovered through automated search by Ramachandran et al. at Google in 2017.

$\text{SiLU}(x) = x \cdot \sigma(x) = \frac{x}{1 + e^{-x}}$

Where:

• $x$ is the input value
• $\sigma(x)$ is the sigmoid function applied to $x$
• The output is the input scaled by its own sigmoid

SiLU has become the default in computer vision: EfficientNet, YOLOv5 through YOLO26, and many diffusion models. It maintains near-zero dead neuron ratios and provides slightly better gradient flow than GELU for convolutional networks.

SwiGLU: The LLM Standard

SwiGLU, introduced by Shazeer (2020), combines Swish with a Gated Linear Unit. It's the activation inside the feed-forward network of LLaMA, PaLM, Gemini, Mistral, and most large language models built after 2022.

$\text{SwiGLU}(x, W_1, W_2, b_1, b_2) = (\text{SiLU}(xW_1 + b_1)) \odot (xW_2 + b_2)$

Where:

• $x$ is the input vector
• $W_1, W_2$ are separate learned weight matrices
• $b_1, b_2$ are bias terms
• $\odot$ is element-wise multiplication
• One path uses SiLU as a gate; the other is a linear transformation

SwiGLU requires a third weight matrix compared to standard feed-forward layers, increasing parameter count by about 50%. To compensate, the hidden dimension is typically reduced by $\frac{2}{3}$, keeping total parameters roughly equal while achieving better performance per parameter.

Softmax Converts Scores to Probabilities

Softmax is the standard output-layer activation for multi-class classification, not used inside hidden layers. In our digit classifier, softmax converts 10 raw logits into a probability distribution over digits 0-9.

$\text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{K} e^{x_j}}$

Where:

• $x_i$ is the raw logit for class $i$
• $K$ is the total number of classes (10 for digit classification)
• $e^{x_i}$ exponentiates each logit, making differences more pronounced
• The denominator normalizes so all outputs sum to 1

Softmax also plays a central role in transformer attention, converting attention scores into weights that sum to 1. The temperature parameter in LLM sampling directly modifies the softmax distribution.

Mish Offers Marginal Gains in Specific Tasks

Mish, proposed by Misra in 2019, gained attention after YOLOv4 adopted it as a smooth, non-monotonic alternative to ReLU.

$\text{Mish}(x) = x \cdot \tanh(\text{softplus}(x)) = x \cdot \tanh(\ln(1 + e^x))$

Where:

• $x$ is the input value
• $\text{softplus}(x) = \ln(1 + e^x)$ is a smooth approximation of ReLU
• $\tanh$ squashes the softplus output to (-1, 1)
• The product with $x$ creates the self-gating behavior

Benchmarks show Mish improves over ReLU by roughly 1-2% on image classification and object detection. It's computationally more expensive than both ReLU and GELU, though. In 2026, Mish occupies a niche role: worth trying in vision pipelines where you've already tuned everything else and want to squeeze out the last fraction of a percent.

Choosing the Right Activation Function

Activation function choice depends on your architecture, not your task. Here's the decision framework.

Click to expandGradient flow comparison showing how different activation functions handle signal propagation through deep networks

When to Use Each Activation

ReLU: Default for CNNs and standard feedforward networks. Pair with He initialization and batch normalization.

Leaky ReLU / PReLU: When you observe dying neurons with standard ReLU, or in deep networks without residual connections.

GELU: Default for transformer encoders (BERT, ViT). Switching from GELU to something else during fine-tuning can degrade performance.

SiLU/Swish: Default for modern CNNs (EfficientNet, YOLO family). Often interchangeable with GELU; the difference is typically within noise.

SwiGLU: Default for LLM feed-forward networks. If you're building or fine-tuning a language model, the architecture likely uses SwiGLU already.

Softmax: Output layer for multi-class classification only.

Sigmoid: Output layer for binary or multi-label classification. Avoid in hidden layers of deep networks.

When NOT to Use Specific Activations

• Sigmoid or tanh in hidden layers of deep networks: Vanishing gradients will cripple training beyond 3-5 layers.
• ReLU in transformers: GELU or SwiGLU consistently outperform ReLU in attention-based architectures.
• GELU in resource-constrained inference: The error function computation adds overhead. ReLU is 2-3x faster per element.
• Mish as a first choice: The computational cost rarely justifies the marginal accuracy gain. Try it last, not first.

Production Considerations

Activation functions affect more than just accuracy:

import torch
import torch.nn as nn

# Running example: digit classifier with different activations
class DigitClassifier(nn.Module):
    def __init__(self, activation="relu"):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 10)

        activation_map = {
            "relu": nn.ReLU(),
            "leaky_relu": nn.LeakyReLU(0.01),
            "gelu": nn.GELU(),
            "silu": nn.SiLU(),
            "mish": nn.Mish(),
            "tanh": nn.Tanh(),
        }
        self.act = activation_map[activation]

    def forward(self, x):
        x = x.view(-1, 784)          # flatten 28x28 image
        x = self.act(self.fc1(x))    # hidden layer 1
        x = self.act(self.fc2(x))    # hidden layer 2
        x = self.fc3(x)              # raw logits (softmax applied in loss)
        return x

# Instantiate with GELU (the modern default for transformers)
model = DigitClassifier(activation="gelu")
print(f"Parameters: {sum(p.numel() for p in model.parameters()):,}")
# Expected output:
# Parameters: 235,146

How Activation Choice Affects Training Dynamics

Activation functions influence three aspects of training that compound over epochs.

Gradient magnitude: Sigmoid and tanh compress gradients into narrow ranges, slowing weight updates in early layers. ReLU maintains unit gradients for positive inputs, and GELU/SiLU maintain near-unit gradients. In our digit classifier, switching from sigmoid to ReLU typically cuts the epochs needed to reach 95% accuracy by half.

Sparsity: ReLU produces truly sparse activations (many exact zeros). GELU and SiLU produce pseudo-sparse activations (many near-zero but not exactly zero). Some sparsity helps generalization, but too much (dying ReLU) is harmful.

Loss surface smoothness: Smooth activations like GELU, SiLU, and Mish create smoother loss surfaces, which helps adaptive optimizers like AdamW find better local minima. The original transformer paper used ReLU, but subsequent work showed measurable improvements from switching to GELU.

Conclusion

Activation functions have evolved from mathematical curiosities to critical architectural decisions. Sigmoid and tanh dominated early neural networks until ReLU made deep learning practical. ReLU's variants (Leaky ReLU, PReLU, ELU) patched its dying neuron problem, but the real shift came with GELU and SiLU, which matched attention-based architectures far better than any piecewise linear function could.

The modern playbook is straightforward: GELU for transformers, ReLU for standard CNNs and feedforward networks, SiLU for modern vision architectures, and SwiGLU for LLMs from scratch. These aren't arbitrary preferences; they're backed by years of empirical evidence across thousands of experiments. Understanding why each activation works (gradient flow, smoothness, sparsity) makes you a better practitioner than memorizing formulas alone.

For deeper context, explore how activation functions interact with backpropagation to shape gradient flow, or see how AdamW and other optimizers compensate for activation-induced gradient patterns. Our guide to building neural networks from scratch puts these activations into practice with working code.

Interview Questions

Q: Why did ReLU replace sigmoid as the default activation in deep networks?

Sigmoid's maximum derivative is 0.25, so gradients shrink exponentially across layers, making deep networks untrainable. ReLU passes gradients of exactly 1.0 for positive inputs, allowing networks with dozens of layers to train effectively. It's also computationally cheaper: a comparison with zero instead of exponential calculations.

Q: What is the dying ReLU problem and how would you fix it?

When a neuron's weighted sum stays consistently negative, ReLU outputs zero with zero gradient, so the neuron permanently stops learning. Fixes include Leaky ReLU (small negative slope of 0.01), PReLU (learned slope), proper He initialization, reducing the learning rate, or switching to GELU/SiLU which never produce exactly zero gradients.

Q: Why do modern transformers use GELU instead of ReLU?

GELU provides smooth, probabilistic gating that aligns with the roughly Gaussian pre-activation distributions produced by attention layers. Unlike ReLU's hard cutoff, GELU transitions smoothly, creating better optimization surfaces. Empirically, GELU outperforms ReLU by 0.5-1.5% on transformer benchmarks, and BERT, GPT-2/3, and ViT all standardized on it.

Q: Explain the difference between SiLU (Swish) and GELU.

Both are smooth, non-monotonic, and self-gating. SiLU uses $x \cdot \sigma(x)$ (sigmoid gate), while GELU uses $x \cdot \Phi(x)$ (Gaussian CDF gate). Their shapes are nearly identical. The practical difference is convention: GELU dominates transformers because BERT standardized it; SiLU is preferred in modern CNNs (EfficientNet, YOLO). Performance differences are typically within noise.

Q: When would you use sigmoid as an activation function in a modern network?

Sigmoid belongs in the output layer for binary classification or multi-label classification (one sigmoid per label, since labels are independent). It's also used internally as a gating mechanism in LSTMs and GRUs. Never use sigmoid in hidden layers of deep networks due to vanishing gradients.

Q: What is SwiGLU and why have modern LLMs adopted it?

SwiGLU combines SiLU with a Gated Linear Unit, splitting the feed-forward layer into a gate path (SiLU) and a value path (linear). Element-wise multiplication gives the model finer control over information flow. LLaMA, PaLM, Gemini, and Mistral all use SwiGLU because it consistently outperforms standard GELU feed-forward layers, though it requires about 50% more parameters per layer (compensated by reducing hidden dimensions).

Q: A colleague suggests using tanh activations in a 20-layer network. What's your response?

Tanh would cause vanishing gradients, making early layers nearly impossible to train. Even though its gradient peaks at 1.0 (vs sigmoid's 0.25), it still saturates for large inputs. I'd recommend ReLU with He initialization and batch normalization, or GELU if the architecture involves attention. If tanh is required, residual connections would be essential to maintain gradient flow.

Q: How does activation function choice affect model quantization for deployment?

ReLU is the most quantization-friendly because its output is zero or positive, needing only unsigned integer representation. GELU and SiLU produce small negative values, requiring signed representation, which slightly reduces effective precision. SwiGLU's extra matrix multiplication can amplify quantization errors. For INT8 deployment, ReLU-based models typically lose less accuracy, though the gap has narrowed with techniques like GPTQ and AWQ.