Let's Data ScienceLEARN • BUILD • STAY AHEAD

LearnAI Evals

AI Evals.
Test, Measure & Ship LLM Apps.

Turn "it feels good" into a measured, regression-gated quality system — error analysis, scorers, LLM-as-judge alignment, confidence intervals, and a CI gate. Model outputs and judge verdicts are baked; every scorer, statistic, and gate you write runs for real on the Helpwell support assistant. Runnable Python, no API keys.

Course Overview

Modules

8

Duration

~9 hours

Helpwell Eval SuiteFaked Outputs, Real MachineryRunnable Python (Pyodide)Scorers → Judge → CI Gate

Module 1 is free. The full 8-module course is part of Pro.

Learning Modules

Each module combines animated explanations, hands-on Python practice, and a knowledge check. You measure ONE thing — the Helpwell support assistant from the RAG and Agents courses — turning vibes into a defensible number, a validated judge, honest error bars, and a gate that blocks a bad ship.

Module 1: Beyond Vibes: What an Eval Actually Is

Vibes don’t ship. Build a real eval — a fixed input set, a definition of "good", and a scorer that returns a number you can defend. The flywheel (analyze → measure → improve → re-measure), offline vs online, and a map of every scorer you’ll build.

Sections

1Why Vibes Don’t Scale (and Benchmarks Lie for Your Product)

2What an Eval Actually Is — Inputs, a Definition of Good, a Scorer

3The Eval Flywheel — Analyze, Measure, Improve, Re-measure

4Offline vs Online — a Closed Loop, Not a Choice

5The Scorer Ladder — a Map of the Course

Module 2: Cheap & Trustworthy: Deterministic Scorers as Classifiers

The cheapest scorer that works. Exact-match and regex (normalize first!), JSON-schema and value validity — then the big idea: your scorer IS a classifier, so grade it with precision/recall/F1 and see why accuracy lies on rare failures.

Sections

1Deterministic Scorers — Exact, Regex, and Normalize First

2Structured-Output Validity — Parse, Schema, Constraints

3Your Scorer IS a Classifier — Confusion Matrix, Precision, Recall, F1

4Why Accuracy Lies on Imbalanced Eval Sets

5Picking the Operating Point — Which Error Is Expensive

Module 3: Words vs Meaning vs Running It: Reference, Semantic & Functional Scorers

Intermediate Pro

When words aren’t enough. ROUGE/BLEU and where they mislead, embedding cosine and BERTScore for meaning, calibrating a threshold (there’s no universal 0.8), and pass@k — run the code, don’t compare the text.

Sections

1Reference-Overlap Metrics — ROUGE, BLEU, and Where They Mislead

2Reference-Free vs Reference-Based — Which Regime per Task

3Semantic Similarity — Cosine and BERTScore’s Token Match

4Calibrating a Similarity Threshold — No Universal 0.8

5Functional Checks and pass@k — Run It, Don’t Compare Text

Module 4: Look At Your Data: Error Analysis & Failure Taxonomies

Intermediate Pro

The highest-ROI hour you’ll spend: reading your own traces. Open-code free-text notes, axial-code them into a named failure taxonomy, count to prioritize, and diagnose with the three gulfs before you fix anything.

Sections

1Error Analysis — Reading Your Data Is the Highest-ROI Activity

2Open Coding — Free-Text Notes on What Went Wrong

3Axial Coding — Group, Name, Count, Prioritize

4Theoretical Saturation — When to Stop Reading

5The Three Gulfs — Comprehension, Specification, Generalization

Module 5: Don’t Trust the Judge — Align It: LLM-as-Judge

Intermediate Pro

An LLM judge is a model you must validate. When to reach for one (binary, not 1–5), align it to humans with Cohen’s kappa (raw agreement lies), spot its biases (verbosity, position, self-enhancement), and iterate the rubric until it agrees.

Sections

1When You Finally Need a Judge — the Last Rung

2Binary Pass/Fail Beats 1–5

3Align the Judge to Humans — Raw Agreement Lies, Use Kappa

4Judge Biases — Verbosity, Position, Self-Enhancement

5Iterate the Judge — Edit the Rubric, Re-measure Alignment

Module 6: Trustworthy Numbers: Noise, Confidence Intervals & Power

One eval number is a coin flip. Put honest error bars on it — the Wilson interval for a pass rate, the bootstrap for any metric, what the bootstrap can’t do, and how many examples is actually enough (power).

Sections

1One Number Is a Coin Flip — Eval Noise & Small-n

2The Wilson Interval for a Pass Rate

3Bootstrap Confidence Intervals for Any Metric

4What the Bootstrap Can and Can’t Do

5Sample Size & Power — How Many Examples Is Enough

Module 7: Comparing Versions & Gating Regressions

Is v2 really better? Paired tests on the same eval set (McNemar — only the disagreements decide), assumption-free permutation/bootstrap, per-slice analysis with multiple-comparison control, and a CI gate that blocks a deploy only on a real regression.

Sections

1Paired vs Unpaired — Same Eval Set Means Paired

2McNemar’s Test — Only the Disagreements Decide

3Permutation & Paired-Bootstrap — the Assumption-Free Comparison

4Per-Slice Analysis & Multiple Comparisons

5The CI Gate — Block a Deploy Only on a Real Regression

Module 8: The Whole System: RAG, Agent, Production & Benchmark Evals

Put it all together. Retrieval metrics (Recall@k, MRR, nDCG), RAG faithfulness and context precision/recall, agent outcome-vs-trajectory and pass^k reliability, production monitoring/drift/cost, and reading public benchmarks without being fooled.

Sections

1Retrieval Evals — Recall@k, Precision@k, MRR, nDCG

2RAG Generation Evals — Faithfulness & Context Precision/Recall

3Agent Evals — Outcome vs Trajectory, and pass^k Reliability

4Online vs Offline — Monitoring, Drift, Guardrails, Cost

5Benchmark Literacy & Assembling Your Own Harness

AI Evals: Test, Measure & Ship LLM Apps | Let's Data Science | Let's Data Science