Backpropagation: The Engine of Deep Learning

Every time you train a neural network, backpropagation is doing the heavy lifting. It's the algorithm that answers the most important question in optimization: how should each weight change to reduce the loss? Without it, deep learning simply wouldn't exist. Since Rumelhart, Hinton, and Williams published their landmark 1986 paper in Nature, backpropagation has remained the backbone of neural network training, powering everything from image classifiers to the large language models behind modern AI assistants.

We'll work through backpropagation from the ground up using one consistent example: a tiny 2-layer network that predicts a single output from two inputs. You'll see the forward pass, loss computation, and backward pass with actual numbers you can verify on paper, then connect that manual math to the automatic differentiation engines that handle the work in practice.

The Chain Rule Powers Everything

Backpropagation is just the chain rule from calculus applied systematically to a computational graph. If a neural network is a sequence of composed functions, the chain rule tells you how to decompose the derivative of the entire composition into a product of local derivatives at each step.

Consider three functions composed together: $y = f(g(h(x)))$. The chain rule says:

$\frac{dy}{dx} = \frac{df}{dg} \cdot \frac{dg}{dh} \cdot \frac{dh}{dx}$

Where:

• $\frac{df}{dg}$ is the derivative of the outermost function with respect to its input
• $\frac{dg}{dh}$ is the derivative of the middle function with respect to its input
• $\frac{dh}{dx}$ is the derivative of the innermost function with respect to the original input $x$

This decomposition is what makes backpropagation efficient. Instead of computing each weight's derivative independently (requiring a separate forward pass each time), you compute all of them in a single backward pass by reusing intermediate results.

Click to expandComputational graph showing forward pass values and backward pass gradients through a two-layer network

A Concrete Network with Real Numbers

The best way to understand backpropagation is to do it by hand. Here's our running example: a 2-layer network with two inputs, one hidden layer (two neurons), and one output neuron. No activation function on the output, and sigmoid activation on the hidden layer.

Network setup:

• Inputs: $x_1 = 0.5$, $x_2 = 0.8$
• Hidden layer weights: $w_1 = 0.3$, $w_2 = -0.1$, $w_3 = 0.5$, $w_4 = 0.2$
• Hidden biases: $b_1 = 0.1$, $b_2 = -0.2$
• Output weights: $w_5 = 0.4$, $w_6 = -0.3$
• Output bias: $b_3 = 0.05$
• Target: $y_{true} = 1.0$

The Forward Pass Computes Predictions

The forward pass pushes data through the network layer by layer, computing each neuron's output. For our network, the hidden layer computes two values:

$z_1 = w_1 \cdot x_1 + w_2 \cdot x_2 + b_1$

Where:

• $z_1$ is the pre-activation value for hidden neuron 1
• $w_1, w_2$ are the weights connecting inputs to this neuron
• $x_1, x_2$ are the input values
• $b_1$ is the bias term

Hidden neuron 1:

• $z_1 = 0.3(0.5) + (-0.1)(0.8) + 0.1 = 0.15 - 0.08 + 0.1 = 0.17$
• $h_1 = \sigma(z_1) = \sigma(0.17) = \frac{1}{1 + e^{-0.17}} = 0.5424$

Hidden neuron 2:

• $z_2 = 0.5(0.5) + 0.2(0.8) + (-0.2) = 0.25 + 0.16 - 0.2 = 0.21$
• $h_2 = \sigma(z_2) = \sigma(0.21) = \frac{1}{1 + e^{-0.21}} = 0.5523$

Output neuron:

• $\hat{y} = w_5 \cdot h_1 + w_6 \cdot h_2 + b_3 = 0.4(0.5424) + (-0.3)(0.5523) + 0.05$
• $\hat{y} = 0.2170 - 0.1657 + 0.05 = 0.1013$

The network predicted 0.1013 when the target is 1.0. That's way off, which means the loss will be large and the gradients will push the weights hard.

Loss Computation Measures the Error

Loss functions quantify how wrong the prediction is. We'll use mean squared error (MSE) for this regression task:

$L = \frac{1}{2}(\hat{y} - y_{true})^2$

Where:

• $L$ is the loss (scalar value we want to minimize)
• $\hat{y}$ is the network's prediction
• $y_{true}$ is the ground truth target
• The $\frac{1}{2}$ factor simplifies the derivative (the 2 from the power rule cancels it)

The Backward Pass Computes Gradients

The backward pass is where backpropagation happens. Starting from the loss, we compute the gradient of the loss with respect to every weight by applying the chain rule backward through the graph.

Step 1: Gradient of loss with respect to the prediction.

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y_{true} = 0.1013 - 1.0 = -0.8987$

Step 2: Gradients with respect to output weights. Since $\hat{y} = w_5 h_1 + w_6 h_2 + b_3$:

$\frac{\partial L}{\partial w_5} = \frac{\partial L}{\partial \hat{y}} \cdot h_1 = (-0.8987)(0.5424) = -0.4875$

$\frac{\partial L}{\partial w_6} = \frac{\partial L}{\partial \hat{y}} \cdot h_2 = (-0.8987)(0.5523) = -0.4963$

$\frac{\partial L}{\partial b_3} = \frac{\partial L}{\partial \hat{y}} \cdot 1 = -0.8987$

Step 3: Gradients flowing back to the hidden layer. This is where the chain rule shines. We need $\frac{\partial L}{\partial h_1}$ and $\frac{\partial L}{\partial h_2}$:

$\frac{\partial L}{\partial h_1} = \frac{\partial L}{\partial \hat{y}} \cdot w_5 = (-0.8987)(0.4) = -0.3595$

$\frac{\partial L}{\partial h_2} = \frac{\partial L}{\partial \hat{y}} \cdot w_6 = (-0.8987)(-0.3) = 0.2696$

Step 4: Through the sigmoid activation. The sigmoid derivative is $\sigma'(z) = \sigma(z)(1 - \sigma(z))$:

$\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial h_1} \cdot \sigma'(z_1) = (-0.3595)(0.5424)(1 - 0.5424) = (-0.3595)(0.2482) = -0.0893$

$\frac{\partial L}{\partial z_2} = \frac{\partial L}{\partial h_2} \cdot \sigma'(z_2) = (0.2696)(0.5523)(1 - 0.5523) = (0.2696)(0.2474) = 0.0667$

Step 5: Gradients with respect to input weights.

$\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_1} \cdot x_1 = (-0.0893)(0.5) = -0.0446$

$\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial z_1} \cdot x_2 = (-0.0893)(0.8) = -0.0714$

$\frac{\partial L}{\partial w_3} = \frac{\partial L}{\partial z_2} \cdot x_1 = (0.0667)(0.5) = 0.0334$

$\frac{\partial L}{\partial w_4} = \frac{\partial L}{\partial z_2} \cdot x_2 = (0.0667)(0.8) = 0.0534$

Click to expandChain rule decomposition showing gradient flow and multiplication through each layer of the network

Computational Graphs Make Backpropagation Systematic

A computational graph represents every operation in the forward pass as a node, with edges showing data dependencies. Backpropagation traverses this graph in reverse topological order, accumulating gradients along every path from the loss to each parameter.

Each node stores its output value (from the forward pass) and its local gradient (for the backward pass). The backward pass applies a simple rule at each node: multiply the incoming gradient by the local gradient and pass the result to all parent nodes.

When a node feeds into multiple downstream operations, gradients from those paths are summed. This is the multivariate chain rule in action.

This perspective also explains why backpropagation is $O(n)$ in the number of operations, matching the forward pass complexity. Each edge is visited exactly once during the backward pass. Numerical differentiation, by comparison, requires $O(p)$ forward passes for $p$ parameters. For a model with billions of parameters, that difference is everything.

Vanishing and Exploding Gradients

Vanishing and exploding gradients are failure modes where gradients become either too small or too large as they propagate backward through many layers.

Why Gradients Vanish

In a deep network with $L$ sigmoid layers, the gradient at each layer gets multiplied by $\sigma'(z) \cdot w$, where $\sigma'(z) \leq 0.25$. After $L$ layers:

$\frac{\partial L}{\partial w^{(1)}} \propto \prod_{l=1}^{L} \sigma'(z^{(l)}) \cdot w^{(l)}$

Where:

• $\frac{\partial L}{\partial w^{(1)}}$ is the gradient at the first layer
• $\sigma'(z^{(l)})$ is the sigmoid derivative at layer $l$, bounded by 0.25
• $w^{(l)}$ is the weight at layer $l$
• The product shrinks exponentially with depth

Why Gradients Explode

The opposite happens when weights are large. If $|\sigma'(z) \cdot w| > 1$ at each layer, the product grows exponentially. Training becomes unstable: weights oscillate wildly, loss spikes to infinity, and NaN values appear in your tensors.

Solutions That Actually Work

Click to expandGradient magnitude comparison across 10 layers showing vanishing gradients with sigmoid versus stable gradients with ReLU

Manual Backpropagation vs. Automatic Differentiation

Nobody implements backpropagation by hand for production networks. Automatic differentiation (autodiff) computes exact gradients by recording the computational graph and applying the chain rule automatically. There are two modes:

Forward mode propagates derivatives from inputs to outputs, computing the Jacobian one column at a time. Efficient when you have few inputs and many outputs.

Reverse mode is what backpropagation uses. It propagates derivatives backward, computing the Jacobian one row at a time. Since neural networks have one scalar loss and millions of parameters, reverse mode is vastly more efficient: one backward pass gives you all gradients.

Modern Autograd: PyTorch, JAX, and torch.compile

Modern frameworks handle backpropagation through sophisticated autograd engines. Here's how the same gradient computation looks in practice.

PyTorch's Dynamic Computational Graph

PyTorch (version 2.10 as of March 2026) builds the computational graph on the fly as operations execute. This "define-by-run" approach means you can use standard Python control flow (if statements, loops) and still get correct gradients.

import torch

# Same network as our running example
x = torch.tensor([0.5, 0.8])
w_hidden = torch.tensor([[0.3, 0.5], [-0.1, 0.2]], requires_grad=True)
b_hidden = torch.tensor([0.1, -0.2], requires_grad=True)
w_out = torch.tensor([0.4, -0.3], requires_grad=True)
b_out = torch.tensor([0.05], requires_grad=True)

# Forward pass
z = x @ w_hidden + b_hidden       # [0.17, 0.21]
h = torch.sigmoid(z)               # [0.5424, 0.5523]
y_hat = h @ w_out + b_out          # [0.1013]

# Loss
target = torch.tensor([1.0])
loss = 0.5 * (y_hat - target) ** 2  # 0.4038

# Backward pass — one call computes ALL gradients
loss.backward()

print(f"dL/dw_out: {w_out.grad}")    # tensor([-0.4875, -0.4963])
print(f"dL/dw_hidden:\n{w_hidden.grad}")
# tensor([[-0.0446,  0.0334],
#         [-0.0714,  0.0534]])

Those gradient values match our hand computation exactly. That's the beauty of PyTorch's autograd engine: it applies the same chain rule math, but without any manual derivatives.

JAX's Functional Approach

JAX (version 0.9.1 as of March 2026) takes a functional approach. You write a pure function and jax.grad returns a new function that computes its gradient:

import jax
import jax.numpy as jnp

def forward(params, x, target):
    w_hidden, b_hidden, w_out, b_out = params
    z = x @ w_hidden + b_hidden
    h = jax.nn.sigmoid(z)
    y_hat = h @ w_out + b_out
    loss = 0.5 * (y_hat - target) ** 2
    return loss.squeeze()

# jax.grad returns a FUNCTION that computes gradients
grad_fn = jax.grad(forward)

params = (
    jnp.array([[0.3, 0.5], [-0.1, 0.2]]),  # w_hidden
    jnp.array([0.1, -0.2]),                  # b_hidden
    jnp.array([0.4, -0.3]),                  # w_out
    jnp.array([0.05]),                        # b_out
)

grads = grad_fn(params, jnp.array([0.5, 0.8]), jnp.array([1.0]))
# grads has the same structure as params — one gradient per parameter

torch.compile Accelerates the Backward Pass

torch.compile (available since PyTorch 2.0) uses TorchDynamo to capture and optimize the entire computational graph, including the backward pass. AOT Autograd traces the backward pass at compile time, enabling kernel fusion:

@torch.compile
def train_step(model, x, target):
    y_hat = model(x)
    loss = torch.nn.functional.mse_loss(y_hat, target)
    loss.backward()
    return loss
# Typical speedup: 1.3x-2x on GPU for medium-to-large models

In benchmarks, torch.compile reduces backward pass time by 30 to 50% on transformer architectures because gradient computations involve many small operations that benefit from kernel fusion.

Common Pitfalls and How to Avoid Them

Real-world backpropagation training runs into specific failure modes beyond vanishing gradients. Here's a quick reference:

When Backpropagation Works and When It Struggles

Backpropagation is the default training algorithm for any differentiable model, but it has clear boundaries.

Use backpropagation when:

• Your model is composed of differentiable operations
• You have a scalar loss function to minimize
• Gradient information is meaningful (smooth loss surface)
• You have enough memory to store activations for the backward pass

Backpropagation struggles when:

• The loss function is non-differentiable (use REINFORCE or straight-through estimators)
• The loss surface is extremely non-convex with many sharp local minima
• Memory is severely constrained (look into gradient checkpointing or reversible architectures)
• You need second-order information (consider L-BFGS or natural gradient methods)

For a deeper treatment of the optimizers that use backpropagation's gradients, see our guide on deep learning optimizers from SGD to AdamW.

Conclusion

Backpropagation is the chain rule applied systematically to a computational graph. That simple idea enables training networks with billions of parameters. The algorithm hasn't fundamentally changed since 1986; what's changed is our ability to execute it efficiently at massive scale.

Our running example produced the exact same gradients by hand and through PyTorch's autograd. Frameworks don't do anything magical. They automate the same math while handling numerical stability, memory management, and GPU parallelism that would be painful to manage manually.

If you're building neural networks from scratch, our guide on building a neural network from scratch in Python walks through forward pass, backpropagation, and weight updates end to end. To understand why different loss surfaces respond differently to gradient-based optimization, the bias-variance tradeoff explains the fundamental tension every model faces.

The best way to truly internalize backpropagation is to compute a few gradients by hand, verify them against autograd, and then trust the framework to handle the rest.

Interview Questions

Q: Walk me through backpropagation in your own words. What's the core idea?

Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule backward through the network's computational graph. Starting from the loss, each node multiplies the incoming gradient by its local derivative and passes the result upstream. The entire set of gradients is computed in a single backward pass with the same time complexity as the forward pass.

Q: Why does backpropagation use reverse mode autodiff instead of forward mode?

Neural networks typically have millions of parameters but only one scalar loss. Reverse mode computes one row of the Jacobian per pass, so a single backward pass gives gradients for all parameters. Forward mode computes one column per pass, meaning you'd need one pass per parameter, which is computationally infeasible for large models.

Q: What causes vanishing gradients, and how do you fix them?

Vanishing gradients occur when the product of local derivatives across many layers shrinks exponentially, often because activations like sigmoid have a maximum derivative of 0.25. The most effective fixes are using ReLU activations (derivative of 1 for positive inputs), adding residual connections (providing a gradient shortcut), and applying proper weight initialization (Xavier or He).

Q: You notice your model's loss suddenly jumps to NaN during training. What's your debugging process?

Check gradient norms before the NaN appears; if they're growing exponentially, apply gradient clipping. Then look for numerical instability: log(0), division by zero, or raw softmax without the max-subtraction trick. Check for corrupted data (missing values that become NaN in tensors), and try reducing the learning rate.

Q: Why does training use more memory than inference?

Training stores all intermediate activations from the forward pass because the backward pass needs them for gradient computation. Inference only keeps the current layer's activations. Gradient checkpointing trades compute for memory by discarding some activations and recomputing them during the backward pass.

Q: What happens if you forget optimizer.zero_grad() in PyTorch?

Gradients accumulate across backward passes because PyTorch adds new gradients to existing .grad tensors rather than replacing them. Without zeroing, each step uses the sum of current and all previous gradients, producing incorrect updates. This behavior is intentional for gradient accumulation strategies that simulate larger batch sizes.

Q: How would you verify that your custom backward pass implementation is correct?

Use numerical gradient checking: for each weight, compute $\frac{L(w + \epsilon) - L(w - \epsilon)}{2\epsilon}$ with a small $\epsilon$ (around $10^{-5}) and compare it to the analytical gradient. The relative difference should be below $10^{-5}. Both PyTorch (torch.autograd.gradcheck) and JAX provide built-in utilities for this verification. Always test with double precision to avoid floating-point noise.

Q: A colleague proposes using sigmoid activations throughout a 50-layer network. What's your response?

Virtually untrainable. Each sigmoid layer multiplies the gradient by at most 0.25, so after 50 layers it's attenuated by $0.25^{50} \approx 10^{-30}$. The first layers receive essentially zero gradient signal. Use ReLU-family activations with residual connections, which is the standard pattern for very deep networks.