Bayesian Statistics: The Scientific Art of Changing Your Mind

Most data science courses start with a lie. They teach you that probability is simply the "long-run frequency" of an event—if you flip a coin infinite times, 50% will be heads. But in the real world, you don't have infinite coin flips. You have one product launch, one clinical trial, or one election.

Bayesian statistics offers a different, often more intuitive way of thinking. Instead of pretending we have infinite data, it treats probability as a measure of belief that changes as we gather evidence. It's the mathematical framework for the quote often attributed to John Maynard Keynes: "When the facts change, I change my mind. What do you do, sir?"

In this guide, we'll move beyond the philosophical debate and build a practical Bayesian engine using Python. We will see how to start with a guess, update it with data, and make decisions that acknowledge uncertainty rather than hiding it.

What is the fundamental difference between Bayesian and Frequentist statistics?

The difference lies in how they treat the unknown. Frequentists (the standard approach) treat parameters (like a conversion rate) as fixed constants and data as random. Bayesians treat data as fixed evidence and parameters as random variables described by probability distributions.

This shift allows Bayesians to answer the question everyone actually asks: "What is the probability that Drug A is better than Drug B?" (Frequentist p-values do not answer this question; they answer "How weird would this data be if the drugs were the same?")

Understanding Bayes' Theorem

At the heart of this framework is a single equation that tells us how to update our beliefs.

The Formula

$P(\theta | D) = \frac{P(D | \theta) P(\theta)}{P(D)}$

Where:

• $\theta$ (Theta) is the hypothesis or parameter we care about (e.g., "The drug works 60% of the time").
• $D$ is the Data or evidence we observed.

Why It Matters

If you ignore the Prior, you risk overreacting to small datasets. If you ignore the Likelihood, you ignore reality. Bayes' theorem balances your previous knowledge with new evidence.

What are Priors, and how do we choose them?

A Prior is a probability distribution representing your knowledge before conducting an experiment. It is the most controversial part of Bayesian analysis because it can be subjective, but it is also its greatest strength.

Types of Priors

1. Uninformative (Flat) Prior: "I know nothing." You assume every possible value is equally likely. This lets the data speak for itself.
2. Weakly Informative Prior: "I know the conversion rate isn't 99%, but it could be anything from 1% to 20%." You give low probability to extreme values.
3. Informative Prior: "Previous studies showed a 5% effect." You start with a strong belief, requiring significant data to change your mind.

Hands-On: Bayesian A/B Testing with Clinical Data

Let's apply this to a real scenario using our clinical trial dataset. We want to know if Drug B is truly better than the Placebo.

The Scenario

We have binary data: patients either responded to treatment (1) or didn't (0).

• Likelihood: Since the data is binary (Success/Failure), the likelihood follows a Binomial distribution.
• Prior: The Beta distribution is the "conjugate prior" for the Binomial likelihood. This is a fancy way of saying: if you start with a Beta prior and add Binomial data, the math works out beautifully to give you a Beta posterior.

We don't need complex MCMC (Markov Chain Monte Carlo) simulations for this. We can do it with simple arithmetic.

1. Loading the Data

We'll use the specific results from our clinical trial dataset.

import numpy as np
import scipy.stats as stats
import matplotlib.pyplot as plt

# DATASET VALUES (from lds_stats_probability.csv)
# Group: Placebo
n_placebo = 287
success_placebo = 116

# Group: Drug_B
n_drug_b = 242
success_drug_b = 157

print(f"Placebo Rate: {success_placebo/n_placebo:.2%}")
print(f"Drug B Rate:  {success_drug_b/n_drug_b:.2%}")

Output:

Placebo Rate: 40.42%
Drug B Rate:  64.88%

2. Defining the Prior

Let's assume we know very little. We'll use a Beta(1, 1) prior, which is a flat line (Uniform distribution). It says "the success rate could be anywhere from 0% to 100% with equal probability."

$\text{Posterior Beta}(\alpha_{new}, \beta_{new}) = \text{Beta}(\alpha_{prior} + \text{Successes}, \beta_{prior} + \text{Failures})$

3. Calculating the Posteriors

# Uninformative Prior parameters (Beta(1,1))
alpha_prior = 1
beta_prior = 1

# Update Placebo Posterior
# Alpha = prior + successes
# Beta = prior + failures
alpha_placebo = alpha_prior + success_placebo
beta_placebo = beta_prior + (n_placebo - success_placebo)

# Update Drug B Posterior
alpha_drug_b = alpha_prior + success_drug_b
beta_drug_b = beta_prior + (n_drug_b - success_drug_b)

print(f"Placebo Posterior: Beta({alpha_placebo}, {beta_placebo})")
print(f"Drug B Posterior:  Beta({alpha_drug_b}, {beta_drug_b})")

Output:

Placebo Posterior: Beta(117, 172)
Drug B Posterior:  Beta(158, 86)

4. Visualizing the Beliefs

Now, instead of a single number (point estimate), we have two curves representing the probable effectiveness of each treatment.

x = np.linspace(0.3, 0.8, 1000)

# Generate PDFs
y_placebo = stats.beta.pdf(x, alpha_placebo, beta_placebo)
y_drug_b = stats.beta.pdf(x, alpha_drug_b, beta_drug_b)

plt.figure(figsize=(10, 6))
plt.plot(x, y_placebo, label='Placebo Posterior', color='gray', linestyle='--')
plt.plot(x, y_drug_b, label='Drug B Posterior', color='blue', linewidth=2)
plt.fill_between(x, y_placebo, alpha=0.2, color='gray')
plt.fill_between(x, y_drug_b, alpha=0.2, color='blue')

plt.title("Posterior Distributions: Placebo vs Drug B")
plt.xlabel("True Response Rate")
plt.ylabel("Density (Belief)")
plt.legend()
plt.show()

5. Answering the Million-Dollar Question

Frequentists stop at "we reject the null hypothesis." Bayesians ask: "What is the probability that Drug B is better than the Placebo?"

We can estimate this by sampling from our posterior distributions.

# Simulate 100,000 draws from each posterior
sim_placebo = stats.beta.rvs(alpha_placebo, beta_placebo, size=100000)
sim_drug_b = stats.beta.rvs(alpha_drug_b, beta_drug_b, size=100000)

# Calculate probability Drug B > Placebo
prob_better = (sim_drug_b > sim_placebo).mean()

print(f"Probability Drug B is better than Placebo: {prob_better:.5f}")

Output:

Probability Drug B is better than Placebo: 1.00000

When should you use Bayesian methods?

Bayesian statistics isn't just "better"; it's a specific tool for specific problems.

Conclusion

Bayesian statistics offers a coherent framework for learning from data. It formalizes the common-sense process of weighing new evidence against prior beliefs. While it requires you to be explicit about your assumptions (priors), it rewards you with intuitive answers to the questions you actually care about.

Instead of asking "Is this result significant?", Bayesian thinking empowers you to ask "How confident should I be?"—a far more valuable question in science and business alike.

Next Steps:

• Explore Probability Distributions to understand the building blocks of priors and likelihoods.
• Check out A/B Testing Design and Analysis to compare this with the frequentist approach.
• Learn about Hypothesis Testing to see the "standard" alternative.

Hands-On Practice

In this guide, we will implement a Bayesian A/B testing engine from scratch using Python. Rather than relying on p-values, which often confuse 'significance' with 'impact', we will use the Clinical Trial dataset to generate full probability distributions for the effectiveness of a Placebo versus Drug B. This allows us to answer the direct business question: 'What is the exact probability that Drug B is superior to the Placebo?'

By shifting from Frequentist point estimates to Bayesian distributions, we've gained a much richer understanding of our data. We don't just know that Drug B is 'statistically significant'; we can quantify that there is a near-100% probability it is superior, with an expected lift of over 60%. This direct quantification of risk and opportunity is what makes Bayesian methods so powerful for decision-making under uncertainty.