Replace “it feels better” with evidence.
Build an evaluation system with failure analysis, deterministic and semantic scorers, aligned LLM judges, uncertainty, regression tests, and release gates.
What you will leave with
You can turn product expectations into an eval dataset and a repeatable gate that blocks a statistically credible regression.
What you will be able to do
The curriculum is organized around these 4 practical outcomes.
Build a failure taxonomy and representative eval set
Write deterministic and semantic scorers
Measure and align an LLM-as-judge
Add uncertainty-aware regression tests and CI gates
Course map
Curriculum
Open a module to inspect every section before you start. Your progress follows you through the course.
Why vibes & benchmarks fail, Input set + good + scorer, The eval flywheel, and more.
Exact / regex / normalize, Schema & value validity, Scorer = classifier, and more.
ROUGE / BLEU / METEOR, Cosine & BERTScore, Threshold calibration, and more.
Read your data, Open coding, Axial coding & counts, and more.
When you need a judge, Binary beats 1–5, Align with kappa, and more.
Eval noise & small-n, Wilson interval, Bootstrap CIs, and more.
Paired comparison, McNemar’s test, Per-slice + corrections, and more.
Retrieval metrics, RAG generation evals, Agent & pass^k, and more.
Who this course is for
Start with the background you have. The prerequisite notes above tell you exactly what is assumed.
AI engineers shipping LLM applications
Quality and platform teams
Technical leaders defining release standards
Continue learning
Each course stands on its own. Pick the one closest to the next task you need to handle.
Build retrieval systems you can measure.
Build the machinery around an agent loop.
Turn one LLM call into a dependable service.
Start the course
Module 1 introduces the language and example used throughout the rest of the course.
