Build a working language model from scratch.
Implement the tokenizer, embeddings, attention, transformer block, decoding, training, and interpretability in NumPy, then run StoryByte in your browser.
Build and inspect a tokenizer
Implement embeddings and causal self-attention
Assemble a transformer block and sampler
Connect training loss and internal activations to generated text
StoryByte, a real roughly one-million-parameter GPT trained on TinyStories and shipped with the course.
Basic Python and comfort with arrays. No PyTorch or GPU is required for the browser course.
Open any module to review its exact sections. Progress and completion follow you through the course.
Topics include autocomplete loop, tokens, byte-pair encoding, and more.
Topics include embedding table, meaning as direction, positional encoding, and more.
Topics include query/key/value, scaled dot-product, the √dₖ scaling, and more.
Topics include the block, feed-forward network, GELU, and more.
Topics include softmax, greedy, temperature, and more.
Topics include next-token prediction, cross-entropy, AdamW, and more.
Topics include logit lens, attention-head patterns, induction-head candidates, and more.
Topics include end-to-end pipeline, parameter budget, documented Llama 2 components, and more.
Engineers who want transformer internals to click
Python learners ready for a serious build
AI practitioners tired of architecture diagrams alone
These are independent LDS courses that extend the closest skills. Choose the direction that matches what you want to build next.
Make a model follow the behavior you need.
Understand what happens between tokens and text.
Engineer what the model sees and returns.
The first module establishes the language and example used throughout the rest of the course.
