Stacking and Blending: The Secret Weapons Behind Winning Competitions

You've tuned every hyperparameter. You've engineered features until your eyes blurred. You've tried Random Forest, XGBoost, and gradient boosting variants, yet your credit scoring model's AUC is stuck at 0.912, exactly 0.003 behind the leaderboard leader. Stacking is how top Kaggle competitors close that gap. Rather than picking a single best model, stacking trains a second-level model that learns how to combine the predictions of multiple diverse base models, canceling out their individual errors and squeezing out performance no single algorithm can reach alone.

Throughout this article, we'll build a credit scoring system that predicts loan defaults. Every formula, every code block, and every diagram references this same dataset so the concepts stay grounded in one concrete scenario.

The Stacking Framework

Stacking (formally called Stacked Generalization, introduced by Wolpert in 1992) is an ensemble learning technique that uses a "meta-model" to learn the optimal way to combine predictions from multiple "base models." It belongs to the broader family of ensemble methods that combine multiple learners to improve predictive performance. Unlike simple majority voting where every model gets equal say, stacking trains a dedicated machine learning model that acts as the final judge, deciding which base model to trust for which types of input.

The CEO Analogy

Think of it as running a lending company. You're the CEO (the meta-learner), and you consult three department heads (the base models) before making every loan decision:

1. The Conservative Analyst (Logistic Regression): Finds linear patterns in income-to-debt ratios.
2. The Pattern Hunter (Random Forest): Catches non-linear interactions between age, employment length, and credit use.
3. The Neighborhood Expert (KNN): Spots applicants who look similar to past defaulters.

For a specific loan application, the Analyst says "approve," the Hunter says "deny," and the Expert says "deny." In a simple vote, "deny" wins 2-1. But as CEO, you've learned from thousands of past decisions. You know that when the Analyst and the Expert disagree on high-income applicants, the Analyst is almost always right. You weigh their inputs dynamically based on context. That process of learning how to weigh the experts is stacking.

The Two-Level Architecture

Click to expandStacking architecture showing training data flowing through K-Fold split to base learners then to meta-learner for final prediction

Stacking operates in two levels:

• Level 0 (Base Learners): A diverse set of models (Random Forest, XGBoost, KNN, SVM) that each independently predict loan default risk.
• Level 1 (Meta-Learner): A simple model (typically Logistic Regression) that takes the predictions of every Level 0 model as input features and produces the final default prediction.

The meta-learner's job isn't complex feature extraction. Its inputs are already highly processed predictions from strong models. It just needs to learn a weighted combination, which is exactly what a linear model excels at.

Preventing Data Leakage with Out-of-Fold Predictions

The most dangerous mistake in stacking is training your base models on the full training set and then using those same models to generate predictions for the meta-learner. The meta-learner would see predictions made on data the base models already memorized, creating massive data leakage. It would learn to blindly trust the base models' overfit predictions rather than their genuine predictive signal.

The fix is K-Fold Cross-Validation to generate "Out-of-Fold" (OOF) predictions. Here's the procedure step by step:

1. Split the training data into $K$ folds (typically 5).
2. Iterate: For each fold $k$ from 1 to $K$:

• Train every base model on the remaining $K-1$ folds.
• Use those trained models to predict on the held-out fold $k$.

3. Stack: Concatenate the predictions from all $K$ held-out folds. You now have a prediction for every training sample, but each prediction was made by a model that never saw that sample during training.
4. Train the meta-learner on these "clean" OOF predictions.

Let's implement this from scratch on our credit scoring dataset to see exactly what happens at each fold.

<!— EXEC —>

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, KFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=15, n_informative=10,
    n_redundant=3, n_classes=2, weights=[0.6, 0.4],
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

kf = KFold(n_splits=5, shuffle=True, random_state=42)
rf = RandomForestClassifier(n_estimators=100, random_state=42)

oof_predictions = np.zeros(len(X_train_scaled))

for fold_idx, (train_idx, val_idx) in enumerate(kf.split(X_train_scaled)):
    X_fold_train = X_train_scaled[train_idx]
    y_fold_train = y_train[train_idx]
    X_fold_val = X_train_scaled[val_idx]

    rf.fit(X_fold_train, y_fold_train)
    oof_predictions[val_idx] = rf.predict(X_fold_val)

    fold_acc = accuracy_score(y_train[val_idx], oof_predictions[val_idx])
    print(f'Fold {fold_idx + 1}: validated on {len(val_idx)} samples, accuracy = {fold_acc:.4f}')

overall_oof_acc = accuracy_score(y_train, oof_predictions)
print(f'\nOverall OOF accuracy: {overall_oof_acc:.4f}')
print(f'OOF predictions shape: {oof_predictions.shape}')

Expected Output:

Fold 1: validated on 320 samples, accuracy = 0.9281
Fold 2: validated on 320 samples, accuracy = 0.9094
Fold 3: validated on 320 samples, accuracy = 0.8938
Fold 4: validated on 320 samples, accuracy = 0.9125
Fold 5: validated on 320 samples, accuracy = 0.8906

Overall OOF accuracy: 0.9069
OOF predictions shape: (1600,)

Fold 1: validated on 320 samples, accuracy = 0.9281
Fold 2: validated on 320 samples, accuracy = 0.9094
Fold 3: validated on 320 samples, accuracy = 0.8938
Fold 4: validated on 320 samples, accuracy = 0.9125
Fold 5: validated on 320 samples, accuracy = 0.8906

Overall OOF accuracy: 0.9069
OOF predictions shape: (1600,)

Each fold validates on 320 samples that the model never trained on. The OOF accuracy (0.9069) is honest, reflecting true generalization. If we'd predicted on the training set directly, we'd see ~0.99+ accuracy, a dangerously misleading number.

Blending: The Simpler Alternative

Blending is a stripped-down version of stacking that replaces K-Fold cross-validation with a single train/holdout split. The concept is identical (train base models, feed their predictions to a meta-learner), but the execution is faster at the cost of wasting data.

The Blending Workflow

1. Split the training data into a training portion (70%) and a blend set (30%).
2. Train all base models on the 70% training portion only.
3. Predict on the 30% blend set to generate features for the meta-learner.
4. Train the meta-learner on those blend-set predictions.

Let's implement blending on our credit scoring data.

<!— EXEC —>

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=15, n_informative=10,
    n_redundant=3, n_classes=2, weights=[0.6, 0.4],
    random_state=42
)
X_train_full, X_test, y_train_full, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_full_scaled = scaler.fit_transform(X_train_full)
X_test_scaled = scaler.transform(X_test)

# Step 1: Split training data into train and blend sets
X_train, X_blend, y_train, y_blend = train_test_split(
    X_train_full_scaled, y_train_full, test_size=0.3,
    random_state=42, stratify=y_train_full
)

print(f'Training set:  {X_train.shape[0]} samples')
print(f'Blend set:     {X_blend.shape[0]} samples')
print(f'Test set:      {X_test_scaled.shape[0]} samples')

# Step 2: Train base models on the training set
base_models = {
    'RF': RandomForestClassifier(n_estimators=100, random_state=42),
    'XGB': GradientBoostingClassifier(n_estimators=100, random_state=42),
    'KNN': KNeighborsClassifier(n_neighbors=7),
}

blend_features = np.zeros((X_blend.shape[0], len(base_models)))
test_features = np.zeros((X_test_scaled.shape[0], len(base_models)))

for i, (name, model) in enumerate(base_models.items()):
    model.fit(X_train, y_train)
    blend_features[:, i] = model.predict_proba(X_blend)[:, 1]
    test_features[:, i] = model.predict_proba(X_test_scaled)[:, 1]
    print(f'{name} trained on {X_train.shape[0]} samples')

# Step 3: Train meta-learner on blend predictions
meta = LogisticRegression()
meta.fit(blend_features, y_blend)
blend_pred = meta.predict(test_features)
blend_acc = accuracy_score(y_test, blend_pred)
print(f'\nBlending ensemble accuracy: {blend_acc:.4f}')

Expected Output:

Training set:  1120 samples
Blend set:     480 samples
Test set:      400 samples
RF trained on 1120 samples
XGB trained on 1120 samples
KNN trained on 1120 samples

Blending ensemble accuracy: 0.9450

Training set:  1120 samples
Blend set:     480 samples
Test set:      400 samples
RF trained on 1120 samples
XGB trained on 1120 samples
KNN trained on 1120 samples

Blending ensemble accuracy: 0.9450

Notice the tradeoff: blending trained each base model once (fast), but the base models only saw 1,120 of the original 1,600 training samples. The remaining 480 were reserved exclusively for the meta-learner. When data is scarce, that 30% loss can hurt.

The Mathematics of Ensemble Diversity

Why does combining models work at all? The answer comes down to variance reduction, and the math reveals a critical requirement: the base models' errors must be uncorrelated.

Suppose we have $M$ models, each with prediction variance $\sigma^2$, and the average pairwise correlation between their errors is $\rho$. The variance of the ensemble's averaged prediction is:

$\text{Var}(\bar{X}) = \rho \sigma^2 + \frac{1-\rho}{M}\sigma^2$

Where:

• $\text{Var}(\bar{X})$ is the variance (expected error) of the ensemble's averaged prediction
• $\rho$ is the average pairwise correlation between the base models' errors
• $\sigma^2$ is the variance of each individual base model
• $M$ is the number of base models in the ensemble

This formula explains everything about stacking strategy. It's why stacking two Random Forests together is nearly useless (high $\rho$ within the same model family), but stacking a Random Forest with Logistic Regression and KNN is powerful (low $\rho$ across different model families).

Let's measure the actual prediction correlations between our base models on the credit scoring data.

<!— EXEC —>

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=15, n_informative=10,
    n_redundant=3, n_classes=2, weights=[0.6, 0.4],
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

models = {
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'Gradient Boosting': GradientBoostingClassifier(n_estimators=100, random_state=42),
    'KNN': KNeighborsClassifier(n_neighbors=7),
    'SVM': SVC(kernel='rbf', probability=True, random_state=42)
}

preds = {}
for name, model in models.items():
    model.fit(X_train_scaled, y_train)
    preds[name] = model.predict(X_test_scaled)

corr_df = pd.DataFrame(preds).corr()
print('Prediction Correlation Matrix:')
print(corr_df.round(4).to_string())

Expected Output:

Prediction Correlation Matrix:
                   Random Forest  Gradient Boosting     KNN     SVM
Random Forest             1.0000             0.8800  0.8565  0.9054
Gradient Boosting         0.8800             1.0000  0.8218  0.8692
KNN                       0.8565             0.8218  1.0000  0.8999
SVM                       0.9054             0.8692  0.8999  1.0000

Prediction Correlation Matrix:
                   Random Forest  Gradient Boosting     KNN     SVM
Random Forest             1.0000             0.8800  0.8565  0.9054
Gradient Boosting         0.8800             1.0000  0.8218  0.8692
KNN                       0.8565             0.8218  1.0000  0.8999
SVM                       0.9054             0.8692  0.8999  1.0000

Look at the numbers. Random Forest and Gradient Boosting correlate at 0.88, which makes sense since both are tree-based ensembles that split data similarly. KNN and Gradient Boosting have the lowest correlation at 0.82, because they approach the problem from fundamentally different angles (distance-based vs. sequential error correction). These are the model pairs where stacking adds the most value.

Full Stacking Implementation with scikit-learn

Now let's bring it all together with scikit-learn's StackingClassifier, which handles the OOF prediction logic automatically.

<!— EXEC —>

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import (
    RandomForestClassifier, GradientBoostingClassifier,
    StackingClassifier
)
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import StandardScaler

np.random.seed(42)
X, y = make_classification(
    n_samples=2000, n_features=15, n_informative=10,
    n_redundant=3, n_classes=2, weights=[0.6, 0.4],
    random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Define base models (Level 0) — maximize diversity
base_models = [
    ('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
    ('xgb', GradientBoostingClassifier(n_estimators=100, random_state=42)),
    ('knn', KNeighborsClassifier(n_neighbors=7)),
    ('svm', SVC(kernel='rbf', probability=True, random_state=42))
]

# Train and evaluate each base model individually
print(f"{'Model':<25} | {'Accuracy':<10}")
print('-' * 38)
for name, model in base_models:
    model.fit(X_train_scaled, y_train)
    pred = model.predict(X_test_scaled)
    acc = accuracy_score(y_test, pred)
    print(f'{name.upper():<25} | {acc:.4f}')

# Build stacking classifier with 5-fold OOF predictions
stacking_clf = StackingClassifier(
    estimators=base_models,
    final_estimator=LogisticRegression(),
    cv=5
)
stacking_clf.fit(X_train_scaled, y_train)
stack_pred = stacking_clf.predict(X_test_scaled)
stack_acc = accuracy_score(y_test, stack_pred)
print(f"{'STACKING ENSEMBLE':<25} | {stack_acc:.4f}")

Expected Output:

Model                     | Accuracy
--------------------------------------
RF                        | 0.9175
XGB                       | 0.9050
KNN                       | 0.9550
SVM                       | 0.9475
STACKING ENSEMBLE         | 0.9500

Model                     | Accuracy
--------------------------------------
RF                        | 0.9175
XGB                       | 0.9050
KNN                       | 0.9550
SVM                       | 0.9475
STACKING ENSEMBLE         | 0.9500

The stacking ensemble scores 0.9500, matching or beating most base models. In this synthetic credit scoring dataset, KNN happens to be strong on its own (0.9550), and the ensemble's meta-learner correctly assigns it high weight. In real-world competitions where no single model dominates across all data slices, the stacking advantage is typically more pronounced. Kaggle winners regularly report 0.5-2% absolute improvement from stacking, which on a 10,000-entry leaderboard can mean the difference between rank 50 and rank 1.

Stacking vs Blending: A Practical Comparison

Click to expandSide-by-side comparison of stacking and blending workflows showing key differences in data usage and computation

Choosing between stacking and blending comes down to your data size and computational budget.

Choosing Base Models for Maximum Diversity

Click to expandConcept map showing how model diversity across families reduces ensemble correlation

The success of any stack depends entirely on the diversity of its base models. The variance reduction formula told us that low $\rho$ is everything. Here's how to achieve it in practice.

The Four Model Families

Pick at least one model from each family to maximize diversity:

Rules of Thumb for Base Model Selection

1. Never stack two models from the same family unless they differ substantially. Two Random Forests with slightly different hyperparameters will have $\rho > 0.95$. That's wasted computation.
2. Vary the feature space. Train one model on all features, another on a PCA-reduced subset, and a third on a hand-selected feature set. Same algorithm, different views of the data.
3. Vary the training data. Train one model on the full training set and another on a bootstrapped sample. This is what bagging already does internally, but you can apply it across your entire stack.
4. Check correlations empirically. Don't guess. Compute the prediction correlation matrix (as we did above) and drop any model that correlates above 0.95 with another.

Advanced Stacking Techniques

Once you've mastered two-level stacking, two extensions can push performance further.

Multi-Layer Stacking

Instead of a single Level 0 followed by a meta-learner, Kaggle Grandmasters sometimes build deeper stacks:

• Level 0: 8-12 diverse base models
• Level 1: 2-3 meta-models, each specializing in a subset (one for tree predictions, one for linear predictions)
• Level 2: A final meta-model combining the Level 1 outputs

This is sometimes called "deep stacking." Each layer reduces error further, but with diminishing returns. In practice, going beyond 3 levels rarely helps and dramatically increases overfit risk.

Feature-Weighted Stacking

In standard stacking, the meta-model only sees the $M$ predictions from the base models. Feature-weighted stacking passes the original features (or a subset) to the meta-learner alongside those predictions. This gives the meta-learner context.

For our credit scoring example, imagine that Model A (Random Forest) is accurate for high-income applicants but weak for low-income ones, while Model B (KNN) shows the opposite pattern. If the meta-learner sees the income feature alongside both predictions, it can learn a conditional weighting rule: "trust Model A when income exceeds $80K; trust Model B otherwise."

# Feature-weighted stacking (conceptual — not EXEC)
# Instead of feeding just [rf_pred, xgb_pred, knn_pred] to the meta-learner,
# feed [rf_pred, xgb_pred, knn_pred, income, credit_score, debt_ratio]
meta_features = np.column_stack([
    oof_rf, oof_xgb, oof_knn,          # base model predictions
    X_train[:, income_idx],              # original income feature
    X_train[:, credit_score_idx],        # original credit score feature
])
meta_model.fit(meta_features, y_train)

When to Use Stacking (and When Not To)

Stacking isn't free. It adds complexity, training time, and overfit risk. Here's a practical decision framework.

Use stacking when:

1. You've already tuned individual models and hit a performance ceiling.
2. Your base models are diverse (prediction correlation < 0.90 between families).
3. The performance gain justifies the complexity. A 0.5% AUC improvement matters in a Kaggle competition; it may not matter in a dashboard prototype.
4. You have enough data. With fewer than 1,000 samples, cross-validation folds become too small for reliable OOF predictions.
5. You're in a competition or high-stakes production scenario (fraud detection, medical diagnosis) where every fraction of a percent counts.

Skip stacking when:

1. A single well-tuned model already meets your performance target.
2. Interpretability is critical. Explaining "the meta-learner weighted the Random Forest at 0.4 and the SVM at 0.35" is hard to sell to regulators.
3. Latency matters. Stacking at inference requires running every base model, then the meta-learner. If your SLA demands sub-10ms predictions, stacking M models may not fit.
4. Your base models aren't diverse. Stacking three gradient boosting variants (XGBoost, LightGBM, CatBoost) helps less than you'd expect since they all partition data with trees.
5. Training compute is constrained. Stacking with $K=5$ folds multiplies base model training time by 5.

Production Considerations

Conclusion

Stacking and blending represent the final stage of model optimization, the point where you've exhausted single-model improvements and start combining diverse perspectives into a unified prediction. The core principle is straightforward: train multiple models that make different kinds of mistakes, then train a meta-learner that figures out which model to trust in which situation. Out-of-fold predictions keep the whole process honest by preventing the meta-learner from seeing data that the base models already memorized.

The mathematics of variance reduction makes diversity non-negotiable. Two models from different families (say, a Random Forest and Logistic Regression) will nearly always beat two models from the same family, regardless of how much you tune the second one. Check the prediction correlation matrix before adding any model to your stack.

For most practitioners, the scikit-learn StackingClassifier with 3-5 diverse base models and a logistic regression meta-learner is the right starting point. If you're working with large-scale data and fast iteration cycles, blending gives you 80% of the benefit at 20% of the compute cost. And if you want to understand the individual models that power a strong stack, explore our guides on XGBoost, Decision Trees, and ensemble methods to build your base learner toolkit.

Diversity is the whole game. A committee of ten clones is useless; a committee of four experts with different training is unstoppable.

Frequently Asked Interview Questions

Q: What is the difference between stacking and bagging?

Bagging (bootstrap aggregating) trains copies of the same algorithm on different bootstrap samples of the data and averages their predictions. Stacking trains different algorithms on the same data and uses a meta-learner to optimally combine them. Bagging reduces variance of a single unstable model; stacking reduces error by exploiting complementary strengths across model families.

Q: Why should the meta-learner be a simple model like logistic regression?

The meta-learner's inputs are just $M$ numbers (one prediction per base model), so the problem it solves is inherently low-dimensional. A complex meta-learner like a neural network would overfit these $M$ features rapidly. A linear model learns the optimal weighted average without memorizing noise, which is exactly what the meta-learner needs to do.

Q: How do you prevent data leakage in stacking?

Use K-Fold cross-validation to generate out-of-fold (OOF) predictions. For each fold, train the base models on the remaining $K-1$ folds and predict on the held-out fold. This ensures every training sample's meta-feature was produced by a model that never saw that sample. Skipping this step causes the meta-learner to learn the base models' overfitting patterns rather than their genuine signal.

Q: When would you choose blending over stacking?

Blending is preferable when dataset size exceeds several million rows (losing 30% still leaves ample training data) or when rapid experimentation matters more than squeezing out final accuracy. It's also simpler to implement and debug. Stacking is better when data is scarce, since it uses 100% of training data through K-Fold rotation.

Q: Your stacking ensemble performs worse than the best individual model. What went wrong?

Three likely causes: (1) Base models aren't diverse enough, meaning their predictions are highly correlated and the meta-learner has nothing to combine. (2) Data leakage in the OOF generation step, causing the meta-learner to overfit. (3) The meta-learner is too complex for the small number of input features. Check the prediction correlation matrix, verify cross-validation is correctly implemented, and simplify the meta-learner.

Q: How many base models should you include in a stack?

Three to five diverse models from different families is the practical sweet spot. Adding a sixth model from a new family can help marginally. Adding a sixth model from a family already represented usually hurts because it increases training time and overfit risk without reducing prediction correlation. Always measure whether each addition actually improves cross-validated performance.

Q: Can you use a stacking ensemble's predictions as features for another stacking ensemble?

Yes, this is called multi-layer stacking. Level 0 generates base predictions, Level 1 generates meta-predictions, and Level 2 combines Level 1 outputs. Each layer must use its own separate cross-validation to avoid leakage. In practice, going beyond 2 layers rarely provides meaningful improvement and significantly increases complexity and overfit risk.

<!— PLAYGROUND_START data-dataset="lds_high_dim" —>

Hands-On Practice

Stacking and Blending are powerful ensemble techniques that push machine learning performance beyond the capabilities of any single algorithm by combining the predictions of multiple diverse models. You'll build a sophisticated Stacking architecture from scratch using the Wine Analysis dataset, learning how a 'meta-learner' can weigh the opinions of different 'base experts' to make superior decisions. This hands-on practice is crucial for understanding how to use the uncorrelated errors of models like KNN, SVM, and Decision Trees to improve robustness and accuracy.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import StackingClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix

# ============================================
# STEP 1: LOAD AND PREPROCESS THE DATA
# ============================================
# We use the Wine dataset which has 13 chemical features and 3 classes (cultivars).
# High dimensionality makes it a good candidate for diverse models to find different patterns.

df = pd.read_csv("/datasets/playground/lds_high_dim.csv")

print(f"Dataset Shape: {df.shape}")
# Expected output: Dataset Shape: (180, 14)

# Separate features and target
X = df.drop('cultivar', axis=1)
y = df['cultivar']

# Stacking works best when base models are diverse.
# Models like KNN and SVM are sensitive to scale, so standardization is critical.
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split into training and testing sets
# We use a stratified split to ensure all wine classes are represented equally
X_train, X_test, y_train, y_test = train_test_split(
    X_scaled, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Training samples: {X_train.shape[0]}, Test samples: {X_test.shape[0]}")
# Expected output: Training samples: 126, Test samples: 54

# ============================================
# STEP 2: DEFINE BASE LEARNERS (LEVEL 0)
# ============================================
# We choose three fundamentally different algorithms:
# 1. KNN (Distance-based)
# 2. Decision Tree (Rule-based/Orthogonal)
# 3. SVM (Geometric/Hyperplane)
# Ideally, their errors should be uncorrelated for Stacking to work well.

base_learners = [
    ('knn', KNeighborsClassifier(n_neighbors=5)),
    ('dt', DecisionTreeClassifier(max_depth=5, random_state=42)),
    ('svm', SVC(kernel='rbf', probability=True, random_state=42))
]

print("\n--- Individual Base Model Performance ---")
# Train and evaluate each base model individually to establish a baseline
for name, model in base_learners:
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    acc = accuracy_score(y_test, y_pred)
    print(f"{name.upper()} Accuracy: {acc:.4f}")

# Expected output (approximate):
# KNN Accuracy: ~0.94
# DT Accuracy: ~0.88
# SVM Accuracy: ~0.98

# ============================================
# STEP 3: BUILD THE STACKING CLASSIFIER (LEVEL 1)
# ============================================
# The 'final_estimator' (Meta-Model) learns how to combine the base learners' predictions.
# We use Logistic Regression as the Meta-Model to linearly weigh the inputs.
# 'cv=5' ensures the meta-model is trained on out-of-fold predictions to prevent leakage.

stacking_model = StackingClassifier(
    estimators=base_learners,
    final_estimator=LogisticRegression(),
    cv=5  # 5-fold cross-validation for generating training data for the meta-model
)

print("\nTraining Stacking Classifier...")
stacking_model.fit(X_train, y_train)

# Evaluate the Stacking Model
y_pred_stack = stacking_model.predict(X_test)
stack_acc = accuracy_score(y_test, y_pred_stack)

print(f"STACKING ENSEMBLE Accuracy: {stack_acc:.4f}")
# Expected output: STACKING ENSEMBLE Accuracy: ~0.98-1.00 (Should match or beat best base model)

print("\nClassification Report (Stacking):")
print(classification_report(y_test, y_pred_stack))
# Expected output:
#               precision    recall  f1-score   support
#            1       1.00      1.00      1.00        18
#            2       1.00      1.00      1.00        21
#            3       1.00      1.00      1.00        15
#     accuracy                           1.00        54

# ============================================
# STEP 4: VISUALIZING WHY STACKING WORKS
# ============================================
# Stacking thrives when models disagree. Let's visualize the correlation of predictions.
# If all models predict exactly the same thing, stacking adds no value.

# Get predictions from base models on the test set
predictions_df = pd.DataFrame()
for name, model in base_learners:
    # Note: We must use the models fitted inside the stacking classifier specifically
    # accessed via stacking_model.named_estimators_[name]
    pred = stacking_model.named_estimators_[name].predict(X_test)
    predictions_df[name] = pred

# Calculate correlation between the PREDICTIONS of different models
corr_matrix = predictions_df.corr()

plt.figure(figsize=(12, 5))

# Plot 1: Prediction Correlation Heatmap
plt.subplot(1, 2, 1)
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', vmin=0, vmax=1)
plt.title('Correlation of Model Predictions')
plt.text(0.5, -0.2, "Lower correlation = Better Stacking potential",
         ha='center', transform=plt.gca().transAxes)

# Plot 2: Model Comparison Bar Chart
plt.subplot(1, 2, 2)
model_names = [name.upper() for name, _ in base_learners] + ['STACKING']
# Recalculate accuracies for plotting
accuracies = []
for name, _ in base_learners:
    acc = accuracy_score(y_test, stacking_model.named_estimators_[name].predict(X_test))
    accuracies.append(acc)
accuracies.append(stack_acc)

bars = plt.bar(model_names, accuracies, color=['skyblue', 'lightgreen', 'salmon', 'gold'])
plt.ylim(0.8, 1.05)
plt.title('Base Models vs Stacking Ensemble')
plt.ylabel('Accuracy')

# Add values on top of bars
for bar in bars:
    height = bar.get_height()
    plt.text(bar.get_x() + bar.get_width()/2., height,
             f'{height:.3f}', ha='center', va='bottom')

plt.tight_layout()
plt.show()

# Interpretation:
# If the correlation between predictions is < 0.9, Stacking usually provides a boost.
# Even if Stacking doesn't drastically improve accuracy, it often improves 'robustness',
# meaning the model is less likely to fail on unseen, noisy data.

Experiment by adding a RandomForestClassifier to the base learners list or changing the final_estimator to a DecisionTreeClassifier to see if a non-linear meta-model performs better. You can also try adjusting the cv parameter in the StackingClassifier; a lower number like 3 might increase variance but speed up training, while 10 provides more solid meta-training data. Finally, observe the correlation heatmap, Stacking is most effective when your base models make different types of errors (low correlation).

<!— PLAYGROUND_END —>