Let's Data ScienceLEARN • BUILD • STAY AHEAD

LearnLLM Foundations

LLM Foundations.
From BPE to Mech Interp.

Eight modules building a complete mental model of how large language models actually work — tokens, embeddings, attention, the transformer block, training, inference, post-training (SFT/DPO/GRPO), and mechanistic interpretability. Each module has an interactive animation you can drive yourself — type text, drag vectors, click attention heads, steer optimizers, sculpt reward landscapes. Lumen Labs builds Lumen-1; you build the understanding.

Course Overview

Modules

8

Duration

~9 hours

BPERoPEMulti-head AttentionSwiGLUGQA / MLAAdamWKV CacheDPOGRPOInduction HeadsSAEs

Prerequisites

Python Fundamentals (recommended) Statistics (helpful for sampling/scaling laws)

Learning Modules

Each module pairs an interactive animation with Pyodide-runnable Python practice and an 11-question knowledge check. Progress saves between sessions.

Tokens — The Words a Model Sees

Why every LLM failure starts here. Walk through BPE byte-by-byte. See why models can't add. Type your own text into the live BPE Forge animation.

Sections

1Why Tokens Exist

2Byte-Pair Encoding (BPE) — The Algorithm

3Vocabulary, Special Tokens, and the Pre-Tokenizer

4Glitch Tokens and Tokenizer Fragility

5Tokenizer Choice in 2026 — Tiktoken vs SentencePiece vs Llama

Embeddings & Positional Encoding

Tokens become vectors. Vectors carry meaning. Positions carry order. RoPE derived from first principles, not hand-waved.

Sections

1Tokens Become Vectors

2The Embedding Table — A Lookup, Not a Computation

3Why Order Matters — The Need for Positional Encoding

4Sinusoidal Positions — The Original Trick

5Rotary Position Embedding (RoPE) — The Modern Default

Attention is Information Routing

The mechanism that powers every modern LLM. Q/K/V, multi-head, causal mask, residual stream. Click any output token in the animation to see who it attends to.

Sections

1The Intuition — Tokens Reading From Each Other

2Q, K, V — Three Lenses on the Same Vector

3The Scaled Dot-Product Attention Formula

4Multi-Head Attention — Subspace Specialization

5Causal Masking and the Residual Stream

The Transformer Block

One block, six modern variants. RMSNorm vs LayerNorm. SwiGLU vs GELU. GQA/MQA/MLA. MoE routing.

Sections

1Anatomy of One Block

2Normalization — LayerNorm, RMSNorm, and Why Order Matters

3The Feed-Forward Network — GELU, SwiGLU, and Width

4Inference-Efficient Attention — MQA, GQA, and MLA

5Mixture of Experts — Routing for Scale

Training at Scale

Cross-entropy, AdamW, scaling laws (Kaplan → Chinchilla → 2026 over-training). FSDP/ZeRO/mixed precision.

Sections

1Next-Token Prediction and Cross-Entropy Loss

2Optimization — AdamW, LR Schedules, Gradient Clipping

3Scaling Laws — Kaplan to Chinchilla and Beyond

42026 Reality — Deliberate Over-Training for Inference Economics

5Distributed Training — FSDP / ZeRO and Mixed Precision

Inference & Decoding

Temperature, top-k, top-p, min-p — sampling demystified. KV cache. Speculative decoding. Test-time compute (o1/o3/R1 scaling).

Sections

1From Logits to Tokens — The Inference Loop

2Sampling Strategies — Temperature, Top-k, Top-p, Min-p

3The KV Cache — Why Inference is Memory-Bound

4Speculative Decoding and Other Production Tricks

5Test-Time Compute — Sequential CoT and Best-of-N

Post-Training — SFT, DPO, GRPO

The 2025-2026 shift. SFT, then DPO (replacing RLHF/PPO), then GRPO + RLVR for reasoning. How o1 and R1 actually work.

Sections

1Why Pre-Training Is Not Enough

2Supervised Fine-Tuning (SFT) — Instruction Following

3Direct Preference Optimization (DPO) — RLHF Without the RL

4Group Relative Policy Optimization (GRPO) and RLVR

5Reasoning Models — o1, R1, and Emergent Chain-of-Thought

Looking Inside & The Frontier

Mechanistic interpretability — induction heads, sparse autoencoders. Modern eval. Mamba, world models, and what comes next.

Sections

1The Residual Stream as Shared Memory

2Induction Heads — The First Real Circuit

3Sparse Autoencoders and Monosemantic Features

4Evaluation — MMLU, HumanEval, lm-eval-harness, Holistic Eval

5The Frontier — Mamba, World Models, and What Comes Next

LLM Foundations — How Large Language Models Actually Work | Free Interactive Course | Let's Data Science