Replace “it feels better” with evidence.
Build an evaluation system with failure analysis, deterministic and semantic scorers, aligned LLM judges, uncertainty, regression tests, and release gates.
Build a failure taxonomy and representative eval set
Write deterministic and semantic scorers
Measure and align an LLM-as-judge
Add uncertainty-aware regression tests and CI gates
The Helpwell assistant, evaluated across LLM, RAG, and agent-style failures using baked outputs and live scoring machinery.
Basic Python and familiarity with an LLM, RAG, or agent workflow.
Open any module to review its exact sections. Progress and completion follow you through the course.
Topics include Why vibes & benchmarks fail, Input set + good + scorer, The eval flywheel, and more.
Topics include Exact / regex / normalize, Schema & value validity, Scorer = classifier, and more.
Topics include ROUGE / BLEU / METEOR, Cosine & BERTScore, Threshold calibration, and more.
Topics include Read your data, Open coding, Axial coding & counts, and more.
Topics include When you need a judge, Binary beats 1–5, Align with kappa, and more.
Topics include Eval noise & small-n, Wilson interval, Bootstrap CIs, and more.
Topics include Paired comparison, McNemar’s test, Per-slice + corrections, and more.
Topics include Retrieval metrics, RAG generation evals, Agent & pass^k, and more.
AI engineers shipping LLM applications
Quality and platform teams
Technical leaders defining release standards
These are independent LDS courses that extend the closest skills. Choose the direction that matches what you want to build next.
Build retrieval systems you can measure.
Build the machinery around an agent loop.
Turn one LLM call into a dependable service.
The first module establishes the language and example used throughout the rest of the course.
