Skip to content
LDS Logo
LDS

A/B Testing & Experimentation

Pro

Causal inference, hypotheses and OEC design, power and sample size, randomization and SRM, CUPED, sequential testing, and multi-armed bandits.

8 modules · Module 1 is free; Modules 2+ require Pro.

View the full course

What this course covers

A module-by-module concept outline. Open the course to learn each topic with animated explanations, in-browser code, practice challenges, and a knowledge check.

Module 1. Why Experiment? The Causal Inference Mindset

Free
Topics
Correlation vs causationThe counterfactualRandom assignmentWhen NOT to test
Sections
  1. 1Correlation Lies (and Dashboards Lie with It)
  2. 2The Counterfactual — The Only Question That Matters
  3. 3Random Assignment — The Most Powerful Tool You Have
  4. 4When NOT to A/B Test
  5. 5Setting Up Bean & Brew's Free-Shipping Test

Module 2. Hypothesis & Metric Design (OEC + Guardrails)

Pro
Topics
Hypothesis writingOEC primary metricSecondary metricsGuardrail metricsLeading vs lagging
Sections
  1. 1Writing a Falsifiable Hypothesis
  2. 2The OEC — Picking the Primary Metric
  3. 3Secondary Metrics — Explaining Mechanism
  4. 4Guardrail Metrics — What You Cannot Break
  5. 5Leading vs Lagging Indicators

Module 3. Statistical Foundations Refresh

Pro
Topics
Null hypothesisp-valuesConfidence intervalsType I / II errorsTwo-sample tests
Sections
  1. 1The Null Hypothesis (and Why It Is Weird)
  2. 2p-values — What They Actually Claim
  3. 3Confidence Intervals — The Better Story
  4. 4Type I & Type II Errors
  5. 5Two-Sample Tests: t-test, z-test, proportion test

Module 4. Sample Size & Power Analysis

Pro
Topics
Statistical powerEffect size (MDE)Sample size formulasDuration trade-offs
Sections
  1. 1The Four Knobs (Effect, Alpha, Power, Variance)
  2. 2Sample Size for Proportions
  3. 3Sample Size for Means and Ratio Metrics
  4. 4The Minimum Detectable Effect (MDE)
  5. 5Trade-offs: Duration vs Detection vs Cost

Module 5. Randomization & Bucketing

Pro
Topics
Random assignmentDeterministic hashingSticky vs session-levelSRM detectionCarryover effects
Sections
  1. 1Why Random Assignment Beats Everything
  2. 2Deterministic Hashing — The Production Pattern
  3. 3Sticky vs Per-Session Assignment
  4. 4SRM — The Sample Ratio Mismatch Canary
  5. 5Carryover Effects & Experiment Isolation

Module 6. Reading Results & Avoiding Pitfalls

Pro
Topics
Peeking problemMultiple testingSimpson's paradoxNovelty & primacySegment analysis
Sections
  1. 1The Peeking Problem (False-Positive Inflation)
  2. 2Multiple Testing Correction (Bonferroni, Benjamini-Hochberg)
  3. 3Simpson's Paradox in Experiments
  4. 4Novelty & Primacy Effects
  5. 5Segment Analysis — Where the Real Story Hides

Module 7. Variance Reduction & Faster Experiments

Pro
Topics
Variance is the enemyCUPEDStratificationSequential testing
Sections
  1. 1Why Variance is the Enemy
  2. 2CUPED — Pre-Experiment Data Saves You
  3. 3Stratification & Post-Stratification
  4. 4Sequential Testing (Peeking, but Legal)
  5. 5Combining CUPED + Sequential — Production Patterns

Module 8. Beyond Standard A/B — The Frontier

Pro
Topics
Multi-armed banditsSwitchback experimentsBayesian A/BQuasi-experimentsCapstone
Sections
  1. 1Multi-Armed Bandits — When Equal Exposure is Wrong
  2. 2Switchback Experiments — Marketplaces & Network Effects
  3. 3Bayesian A/B — The Probability-of-Win Framing
  4. 4Quasi-Experiments — When You Cannot Randomize
  5. 5Capstone — Bean & Brew Free-Shipping Test End-to-End

Ready to start A/B Testing & Experimentation?

Module 1 is free. Unlock the full course with Pro.

Go to the course

All course syllabi

SQL FundamentalsSQL MasterySQL Window FunctionsPython FundamentalsPandas FundamentalsStatistics for Data ScienceProbability for Data ScientistsMachine Learning Fundamentals (scikit-learn)LLM Foundations