Let's Data ScienceLEARN • BUILD • STAY AHEAD

LearnBuild a Tiny LLM

Build a Tiny LLM.
From Tokens to Text.

Build a working language model from scratch, in your browser: the tokenizer, embeddings, attention, the transformer block, and the sampler, in runnable NumPy. Then watch StoryByte, a real ~1M-parameter GPT we trained from scratch on TinyStories, write an original story. Every piece that computes runs live and reproduces the real model. No PyTorch, no GPU, no magic.

Course Overview

Modules

8

Duration

~6.5 hours

StoryByte, a real 1M-param GPTBuild it in NumPyRuns live in your browserTokens → Attention → Text

All 8 modules are free. Sign in to start learning and track your progress, and earn a certificate when you finish.

Learning Modules

Each module combines animated explanations, hands-on NumPy practice, and a knowledge check. You build ONE thing, StoryByte, a real tiny GPT, one piece per module, tracing the same prompt ("Once upon a time") from raw text all the way to a generated story.

Module 1: What an LLM Really Is, and How Text Becomes Numbers

An LLM is autocomplete at scale: guess the next token, append, repeat. Meet StoryByte, then build its first component from scratch, a byte-pair-encoding tokenizer, and see exactly why it writes essays yet can’t count the r’s in "strawberry".

Sections

1Autocomplete, Scaled, The Whole Idea

2Why a Model Can't Read Letters, Tokens

3Build BPE: Most-Frequent-Pair → Merge → Repeat

4Byte-Level BPE, 256 Bytes, Nothing Unknown

5Encode, Decode, and the Strawberry Problem

Module 2: From Tokens to Meaning: Embeddings & Position

Turn token IDs into vectors of meaning. Watch StoryByte place cat next to dog and red next to blue all on its own, prove attention is order-blind with the shuffle test, add positional encoding, and meet the residual stream the rest of the course rides on.

Sections

1The Embedding Table, A Lookup, Not a Computation

2Directions Carry Meaning

3Why Attention Is Order-Blind (The Shuffle Test)

4Injecting Position, Learned, Sinusoidal, RoPE

5The Residual Stream, The Shared Conveyor Belt

Module 3: Attention: How Words Read Each Other

IntermediateFree

The idea that made modern AI. Build scaled dot-product attention from scratch, query/key/value, the √dₖ scaling, causal masking, multi-head, then inspect StoryByte’s real attention maps, the same view interpretability researchers use.

Sections

1The Intuition, Query, Key, Value

2Scaled Dot-Product Attention

3Why Divide by √dₖ, The Volume Knob

4Causal Masking, Blindfold the Future

5Multi-Head Attention, A Committee of Readers

Module 4: The Transformer Block: Stacking the Machine

IntermediateFree

Wrap attention and a feed-forward network (MLP + GELU) in LayerNorm and residual connections to form one reusable block, then stack it. Watch StoryByte’s real prediction sharpen layer by layer before it’s read out into logits.

Sections

1Anatomy of One Block, Two Stations

2The Feed-Forward Network, Per-Word Thinking

3Residuals + LayerNorm, and Why Pre-LN

4Stacking N Blocks, Understanding in Layers

5Final LayerNorm + Unembedding → Logits

Module 5: Making It Speak: From Logits to Words

IntermediateFree

A pile of logits isn’t text. Turn the distribution into words with softmax, greedy (and why it loops), temperature, and top-k / top-p, then drive StoryByte’s real generations and watch it write a story one real token at a time.

Sections

1Logits → Softmax → Probabilities

2Greedy Decoding (and Why It Repeats)

3Temperature, The Creativity Dial

4Top-k vs Top-p, A Smarter Shortlist

5The Decode Loop, Generate StoryByte's First Story

Module 6: How It Learned: Training a Tiny LLM

No teacher labelled a thing. See the one game (predict the next token), cross-entropy and perplexity, and StoryByte’s real loss curve fall from random guessing to fluent storytelling, plus scaling laws and the TinyStories twist.

Sections

1The One Game: Predict the Next Token

2Cross-Entropy = Surprise (and Perplexity)

3The Training Loop, Batch, Loss, Update

4Why Scale Helps, and How to Spend It

5The Twist: Data Beats Size, How We Trained StoryByte

Module 7: Opening the Box: What's Actually Inside

StoryByte is small enough to inspect. Use real interpretability tools, the logit lens, induction heads, superposition, the FFN as memory, on its actual weights, while keeping the discipline of evidence over mind-reading.

Sections

1The Logit Lens, Reading a Hunch Become an Answer

2Attention-Head Roles → Induction Heads

3Features vs Neurons, Superposition

4The FFN as Key-Value Memory

5Honest Narration, Evidence, Never Mind-Reading

Module 8: The Whole Machine & The Frontier

Step back and see it all: the full pipeline end to end, where StoryByte’s 1.09M parameters actually live, how frontier models like GPT-4 differ (less than you’d think), an honest scorecard, and how to train your own.

Sections

1The Full Pipeline, End to End

2Where the Parameters Live

3How Modern LLMs Differ from StoryByte

4The Honest Scorecard, What It Can and Cannot Do

5Where to Go Next

Build a Tiny LLM: From Tokens to Text | Let's Data Science