Build a working language model from scratch.
Implement the tokenizer, embeddings, attention, transformer block, decoding, training, and interpretability in NumPy, then run StoryByte in your browser.
What you will leave with
You finish with a working mental and code model of GPT-style generation, from raw text to one predicted token at a time.
What you will be able to do
The curriculum is organized around these 4 practical outcomes.
Build and inspect a tokenizer
Implement embeddings and causal self-attention
Assemble a transformer block and sampler
Connect training loss and internal activations to generated text
Course map
Curriculum
Open a module to inspect every section before you start. Your progress follows you through the course.
autocomplete loop, tokens, byte-pair encoding, and more.
embedding table, meaning as direction, positional encoding, and more.
query/key/value, scaled dot-product, the √dₖ scaling, and more.
the block, feed-forward network, GELU, and more.
softmax, greedy, temperature, and more.
next-token prediction, cross-entropy, AdamW, and more.
logit lens, attention-head patterns, induction-head candidates, and more.
end-to-end pipeline, parameter budget, documented Llama 2 components, and more.
Who this course is for
Start with the background you have. The prerequisite notes above tell you exactly what is assumed.
Engineers who want transformer internals to click
Python learners ready for a serious build
AI practitioners tired of architecture diagrams alone
Continue learning
Each course stands on its own. Pick the one closest to the next task you need to handle.
Make a model follow the behavior you need.
Understand what happens between tokens and text.
Engineer what the model sees and returns.
Start the course
Module 1 introduces the language and example used throughout the rest of the course.
