SHAP and LIME: Making AI Models Explainable

A bank denies your loan. The decision came from a gradient boosted model with 500 trees. You're owed an explanation — legally, ethically, and practically. SHAP and LIME are the two methods that make this possible, and the EU AI Act's August 2026 compliance deadline makes explainability a legal obligation for high-risk AI systems.

Model interpretability has shifted from "nice to have" to mandatory. The EU AI Act's Article 13 requires high-risk AI systems to provide sufficient transparency for deployers to interpret outputs. The US Fair Credit Reporting Act requires adverse action notices. Beyond compliance, teams that can explain their models debug them faster, catch biased features earlier, and build stakeholder trust more effectively. SHAP 0.51.0 (released March 4, 2026) and LIME remain the dominant tools for this work.

This article uses a single running example throughout: a loan approval model trained on income, credit score, debt ratio, employment years, and loan amount. Every formula, visualization, and code block connects back to this scenario.

Why Black-Box Models Need Explanation

A model's test accuracy tells you how often it's right. It says nothing about why. Two critical problems arise from opacity.

Feature leakage hides inside black boxes. A loan model that seems highly accurate might be secretly using ZIP code as a proxy for race. Feature importance alone doesn't reveal how that feature affects predictions. Until you can see which features drive individual decisions, and in which direction, you can't catch it.

Debugging requires root causes. When your model performs poorly on a specific segment — say, applicants with self-employment income — you need to see what the model is actually doing with their features, not just that accuracy dropped. Interpretability tools give you that window.

There are two ways to think about explanations: global and local. A global explanation describes what the model does on average across all predictions. A local explanation describes why the model made one specific prediction. Both matter in practice. A regulator wants global behavior; an applicant wants their individual decision explained.

There's also an important distinction between model-specific and model-agnostic methods. TreeSHAP is specifically designed for decision trees and gradient boosted models, running in polynomial rather than exponential time. LIME is model-agnostic: it treats any model as a black box and works regardless of what's inside.

LIME: Local Approximation of Any Model

LIME (Local Interpretable Model-agnostic Explanations) was introduced by Ribeiro, Singh, and Guestrin in their 2016 paper "Why Should I Trust You?" The core idea is elegant: you don't need to understand a complex model globally to explain one prediction locally.

For any single prediction, LIME creates a simple surrogate model — usually linear regression — that approximates the black box in a small neighborhood around that point.

Click to expandLIME local approximation process for a single loan applicant prediction

How LIME Works Step by Step

Take our loan applicant with income $72,000, credit score 710, debt ratio 0.35, employment 4 years, and loan amount 18,000 dollars. LIME explains this individual prediction using four steps.

Step 1: Perturbation. LIME creates n synthetic neighbors by randomly sampling feature values near the instance. For a tabular model, this means drawing values from each feature's marginal distribution — sometimes keeping income at $72,000, sometimes setting it to 45,000, sometimes to 90,000.

Step 2: Black-box predictions. Each synthetic neighbor gets passed through your real model. The gradient boosted model (or random forest, neural network — anything) outputs a probability for each neighbor.

Step 3: Proximity weighting. Neighbors closer to the original instance in feature space get higher weight. LIME uses an exponential kernel: weight = exp(-d(x, z)^2 / sigma^2), where d is distance and sigma controls the neighborhood size.

Step 4: Fit a weighted linear model. With n weighted neighbors and their predictions, LIME fits a linear regression. The coefficients of that regression are the explanation — they describe how much each feature pushed the prediction up or down, locally.

The LIME Objective Function

$\xi(x) = \arg\min_{g \in G} \; \mathcal{L}(f, g, \pi_x) + \Omega(g)$

Where:

• $\xi(x)$ is the explanation function for instance $x$
• $G$ is the class of interpretable models (linear models, decision trees)
• $\mathcal{L}(f, g, \pi_x)$ is the fidelity loss — how poorly $g$ approximates $f$ in the local neighborhood defined by proximity measure $\pi_x$
• $\Omega(g)$ is the complexity of the explanation model (number of features)
• $f$ is the original black-box model

LIME for the Loan Applicant

from lime import lime_tabular
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier

# X_train, X_test, y_train are prepared:
# features: income, credit_score, debt_ratio, employment_years, loan_amount
explainer = lime_tabular.LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=X_train.columns.tolist(),
    class_names=['denied', 'approved'],
    mode='classification'
)

# Explain one applicant
instance = X_test.iloc[0]
explanation = explainer.explain_instance(
    data_row=instance.values,
    predict_fn=model.predict_proba,
    num_features=5,
    num_samples=1000  # Higher = more stable explanations
)

for feat, weight in explanation.as_list():
    print(f"{feat}: {weight:+.3f}")
# credit_score > 700: +0.23
# debt_ratio > 0.40: -0.18
# income <= 50000: -0.12
# employment_years <= 3: -0.08
# loan_amount > 20000: -0.04

Each entry is the linear coefficient of that feature condition in the local surrogate model — positive values push toward approval, negative values push toward denial.

SHAP: Game Theory Meets Feature Attribution

SHAP (SHapley Additive exPlanations) was introduced by Lundberg and Lee in their 2017 NeurIPS paper. Where LIME approximates locally, SHAP computes exact feature contributions grounded in cooperative game theory.

The insight: treat a model prediction as a "game" where the players are features, and the "payout" is the prediction value. Shapley values (invented by Lloyd Shapley in 1953 — he won the Nobel Prize in Economics in 2012 partly for this work) fairly distribute the payout among players.

The Shapley Value Formula

$\phi_i(f) = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|!\,(|F|-|S|-1)!}{|F|!} \left[f(S \cup \{i\}) - f(S)\right]$

Where:

• $\phi_i(f)$ is the Shapley value for feature $i$ — its fair contribution to the prediction
• $F$ is the set of all features
• $S$ is a subset of features that does NOT include feature $i$
• $|S|$ is the number of features in subset $S$
• $|F|$ is the total number of features
• $f(S \cup \{i\})$ is the model's prediction when feature $i$ is included with subset $S$
• $f(S)$ is the model's prediction using only the features in $S$ (feature $i$ excluded)
• The fraction $\frac{|S|!\,(|F|-|S|-1)!}{|F|!}$ weights each ordering by how many times that arrangement occurs

The SHAP Efficiency Property

This is what separates SHAP from simpler attribution methods. SHAP values satisfy three axioms:

The efficiency property is particularly useful: it means every SHAP waterfall plot is an exact decomposition. The base value (average prediction across training data) plus all SHAP values sum to exactly the model's output for that instance.

Click to expandSHAP waterfall diagram showing feature contributions summing to the final loan approval probability

SHAP Variants: Choosing the Right Explainer

Directly computing Shapley values requires iterating over all subsets of features — exponential complexity. In practice, four approximations cover virtually every use case.

TreeSHAP is the one you'll use most. Lundberg et al. showed in their 2020 paper that tree Shapley values can be computed exactly in polynomial time by exploiting the tree structure. For a forest with $T$ trees, depth $D$, and max leaf size $L$, TreeSHAP runs in $O(TLD^2)$ — fast enough for production inference.

LinkedIn's FastTreeSHAP (available since 2022, refined in recent releases) pushes TreeSHAP further with two algorithmic variants. FastTreeSHAP v1 is 1.5x faster than the original with the same memory footprint. FastTreeSHAP v2 is 2.5x faster at the cost of slightly more memory — the library automatically selects v2 when your dataset is large enough.

Using SHAP with a Gradient Boosted Model

import shap
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

# Loan dataset setup (shap 0.51.0, xgboost 2.1+)
np.random.seed(42)
n = 1000
X = pd.DataFrame({
    'income': np.random.normal(65000, 20000, n),
    'credit_score': np.random.normal(680, 80, n),
    'debt_ratio': np.random.uniform(0.1, 0.6, n),
    'employment_years': np.random.exponential(5, n),
    'loan_amount': np.random.normal(15000, 5000, n)
})

log_odds = (X['credit_score'] - 680) / 80 * 1.5 + \
           (X['income'] - 65000) / 20000 * 0.8 - \
           X['debt_ratio'] * 2.0 + X['employment_years'] * 0.05
prob = 1 / (1 + np.exp(-log_odds))
y = (prob > np.random.uniform(0, 1, n)).astype(int)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = xgb.XGBClassifier(n_estimators=100, max_depth=4, random_state=42)
model.fit(X_train, y_train)

# TreeSHAP — exact Shapley values for tree models
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# Global feature importance: beeswarm or bar
shap.summary_plot(shap_values, X_test, plot_type="bar")

# Local explanation: waterfall plot for one applicant
shap.waterfall_plot(shap.Explanation(
    values=shap_values[0],
    base_values=explainer.expected_value,
    data=X_test.iloc[0]
))

# Dependence plot: how credit_score interacts with debt_ratio
shap.dependence_plot("credit_score", shap_values, X_test,
                     interaction_index="debt_ratio")

The shap_values array has shape (n_test, n_features). Each row is one prediction; each column is one feature's Shapley value for that prediction. The global importance chart — beeswarm or bar — aggregates across all rows to show which features matter most overall.

SHAP Visualization Types

Beeswarm plot: Every dot is one prediction. Position on the x-axis is the SHAP value; color is the feature's actual value. A cloud of red dots (high feature value) on the right side of the x-axis means high values of that feature push predictions up. This is the most information-dense SHAP visualization.

Waterfall plot: A single-instance breakdown. Starts at the base value (average model output), then stacks each feature's SHAP contribution — up for positive, down for negative — until reaching the final prediction. Perfect for explaining individual decisions to applicants or regulators.

Dependence plot: Shows the relationship between one feature's actual values and its SHAP values across all instances. If the line is monotone, the feature has a simple effect. Kinks and color variation (from the automatically chosen interaction feature) reveal non-linear effects and interactions.

Force plot: An inline waterfall, showing features as colored arrows pushing the prediction from left (base value) to right (final prediction). Useful in dashboards and interactive tools where you embed explanations alongside predictions.

SHAP Interaction Values

Regular SHAP values capture each feature's total contribution to a prediction. But sometimes two features jointly create an effect that neither produces alone. That's what SHAP interaction values capture.

For our loan model: imagine credit score and debt ratio have a combined effect. A good credit score partially offsets a high debt ratio — the model grants some leniency to established borrowers. Regular SHAP values split this interaction somewhat arbitrarily between the two features. SHAP interaction values make it explicit.

The interaction value matrix for a prediction has shape (n_features, n_features). Diagonal entries are the main effects (equivalent to regular SHAP values minus interactions). Off-diagonal entry (i, j) captures how much features $i$ and $j$ jointly influence the prediction beyond their individual effects.

# SHAP interaction values (TreeSHAP only — computationally heavier)
explainer = shap.TreeExplainer(model)
shap_interaction_values = explainer.shap_interaction_values(X_test)
# Shape: (n_test, n_features, n_features)

# Interaction between credit_score and debt_ratio
feature_names = list(X_test.columns)
cs_idx = feature_names.index('credit_score')
dr_idx = feature_names.index('debt_ratio')

# Average interaction strength across all test instances
mean_interaction = np.abs(shap_interaction_values[:, cs_idx, dr_idx]).mean()
print(f"credit_score × debt_ratio interaction: {mean_interaction:.4f}")

# Visualize all pairwise interactions
shap.summary_plot(shap_interaction_values, X_test)

SHAP interaction values are particularly useful in credit risk and healthcare: knowing that "debt_ratio matters more when employment_years is low" is actionable insight a single-feature importance chart can't surface.

Counterfactual Explanations with DiCE

SHAP tells you what drove a prediction. Counterfactual explanations tell you what would have to change to get a different outcome. That's a fundamentally different — and often more useful — question from the applicant's perspective.

DiCE (Diverse Counterfactual Explanations), developed at Microsoft Research, generates multiple realistic counterfactuals: "You would have been approved if your income were $78,000 (up from $65,000)" or "You would have been approved if your debt ratio were 0.28 (down from 0.40)." The diversity is key — showing applicants one path forward, when multiple paths exist, can feel arbitrary.

import dice_ml
from dice_ml import Dice

# DiCE setup for our loan model
data = dice_ml.Data(
    dataframe=pd.concat([X_train, y_train.rename('approved')], axis=1),
    continuous_features=['income', 'credit_score', 'debt_ratio',
                         'employment_years', 'loan_amount'],
    outcome_name='approved'
)
model_for_dice = dice_ml.Model(model=model, backend="sklearn")
exp = Dice(data, model_for_dice, method="random")

# Generate 3 counterfactuals for the denied applicant
denied_instance = X_test.iloc[[0]]  # Our applicant: denied
cf = exp.generate_counterfactuals(
    denied_instance,
    total_CFs=3,
    desired_class="opposite"
)
cf.visualize_as_dataframe()

DiCE-Extended (2025 paper by Broelemann et al.) addresses a known limitation of the original: generated counterfactuals can be fragile, flipping the outcome for the example shown but not for nearby points. DiCE-Extended adds multi-objective optimization to ensure counterfactuals remain valid across small perturbations.

The practical difference between SHAP and DiCE is the question they answer:

SHAP vs LIME: When to Use Which

Both methods explain individual predictions for any model, but they make different trade-offs.

Click to expandSHAP vs LIME comparison of foundation, scope, consistency, and speed

Use SHAP when:

• You're working with tree-based models (XGBoost, LightGBM, Random Forest, CatBoost) — TreeSHAP is fast and exact
• You need both global and local explanations from one framework
• Consistency between explanations matters (regulatory reporting, audits)
• You want to understand feature interactions, not just main effects
• You're doing feature selection based on explanations

Use LIME when:

• Your model is a neural network or SVM and you don't have time for KernelSHAP
• You need a quick, rough local explanation for debugging
• You're working in a real-time serving environment where KernelSHAP latency is unacceptable
• You need explanations for text or image models without the full SHAP ecosystem setup

Production Applications of Explainability

Explanations aren't just for understanding — they actively improve models in production.

Feature Selection with SHAP

SHAP global importance is strictly better than the built-in feature_importances_ in scikit-learn. Sklearn's MDI importance (mean decrease in impurity) is biased toward high-cardinality features and doesn't account for interaction effects. SHAP mean absolute values are unbiased and scale correctly with the feature's actual effect on output.

# SHAP-based feature selection (display-only — requires shap + xgboost)
shap_importance = pd.DataFrame({
    'feature': X_train.columns,
    'shap_importance': np.abs(shap_values).mean(axis=0)
}).sort_values('shap_importance', ascending=False)

# Remove features with near-zero SHAP importance
features_to_keep = shap_importance[
    shap_importance['shap_importance'] > 0.005
]['feature'].tolist()

Bias Detection with SHAP Dependence Plots

A dependence plot for ZIP_code (or any feature correlated with a protected class) will reveal if and how that feature affects predictions. If SHAP values for ZIP code vary systematically by the demographics encoded in those ZIP codes, that's a signal your model may be encoding discriminatory behavior.

The EU AI Act and US fair lending laws both require documentation of this analysis. SHAP makes the required artifact — a chart showing each feature's effect on predictions — straightforward to produce.

Click to expandXAI production pipeline: from model training through SHAP deployment to EU AI Act audit and drift monitoring

Cloud-Native Explainability at Scale

Both major cloud providers have built SHAP into their ML platforms. AWS SageMaker Clarify uses a scalable KernelSHAP approximation that distributes the feature subset sampling across compute instances, making it practical for models with hundreds of features and millions of predictions. Google Vertex AI Explainable AI integrates SHAP for tabular models and Integrated Gradients for image and text models.

The production pattern that's emerged at scale: compute SHAP values on a sample of predictions (say, 10% of daily traffic) and store them alongside predictions. Use the stored SHAP values for drift detection — if the distribution of SHAP values for credit_score shifts week over week, the model's relationship with that feature is changing, even if overall accuracy hasn't dropped yet.

Model Debugging with Local Explanations

When a production model starts behaving strangely on a specific cohort, LIME and SHAP let you look at individual mispredictions directly. Rather than asking "why is accuracy low on self-employed applicants?", you can ask "what does the model see when it looks at this specific applicant who should have been approved?". The waterfall plot answers that question concretely.

LLM and Deep Learning Interpretability

SHAP and LIME both work on neural networks, but the tooling landscape is distinct from tabular models.

For transformer-based language models, the 2024-2025 period produced meaningful advances. TokenSHAP (Moshe and Barkan, 2024) applies Shapley value estimation to individual tokens in an LLM prompt. It treats the LLM as a black box and estimates each token's Shapley value using Monte Carlo sampling. In evaluations, TokenSHAP significantly outperformed attention visualization at identifying which tokens were genuinely causing specific model outputs — attention scores showed which tokens the model was looking at, not which tokens were causally responsible.

Attention visualization examines attention weights between tokens and is the most accessible approach. Its known limitation: high attention weight doesn't guarantee high causal influence. A token the model attends to heavily might not change the output much if removed.

Probing classifiers train small linear classifiers on a frozen layer's representations to test what linguistic information is encoded at each layer. If a probing classifier can predict part-of-speech tags from layer 4 representations with 95% accuracy, layer 4 encodes syntactic information.

Concept bottleneck models (Koh et al., 2020) structure the model itself around human-interpretable concepts. Instead of predicting output directly from input features, the model first predicts a set of human-defined concept scores (e.g., "this X-ray shows consolidation: 0.87"), then predicts the final label from those concepts. Every prediction is inherently explained by the concept scores. The limitation: you need human-defined concepts up front, which is practical in structured domains (radiology, credit) but not for open-domain text.

For simpler deep learning models (CNNs, dense networks), SHAP's DeepExplainer uses a linearization approach based on backpropagation through the network. GradientExplainer uses expected gradients. Both are available in shap 0.51.0.

Understanding how backpropagation works is essential background for gradient-based attribution methods — DeepSHAP and Integrated Gradients both trace gradients through the network in ways that parallel the chain rule.

EU AI Act Compliance: What SHAP Actually Gets You

The EU AI Act's transparency requirements took full effect in stages, with the high-risk AI system obligations applying from August 2026. Article 13 requires that high-risk AI systems be designed to enable deployers to interpret outputs and use them appropriately. This doesn't mandate SHAP specifically — but it does mandate the capability SHAP provides.

Practically, what counts as compliance:

Audit trails. For each high-stakes decision, you need a logged explanation that can be retrieved later. SHAP waterfall values stored alongside predictions satisfy this requirement. LIME explanations can also serve, but SHAP's consistency guarantees make them easier to defend.

Global model documentation. The Act requires providers to document the model's logic in a way that a non-technical auditor can understand. SHAP beeswarm plots and feature importance rankings are the standard artifact for this. They show, at a population level, which features the model relies on and in what direction.

Feature effect documentation. SHAP dependence plots — one per feature, showing how feature values map to SHAP contributions — provide exactly the per-feature documentation the Act's technical documentation requirements anticipate.

The Act explicitly does not ban black-box models, and it does not require models to be inherently interpretable. Post-hoc explanation with SHAP is fully compliant, as long as explanations are stored and accessible.

Limitations of SHAP and LIME

Knowing where these methods break is as important as knowing how to use them.

SHAP assumes feature independence. The Shapley value formula marginalizes over all feature subsets by replacing absent features with their expected values. If features are correlated — and in real loan data, income and employment years certainly are — the "expected value with feature absent" is a counterfactual that may not be realistic. TreeSHAP has a feature_perturbation="tree_path_dependent" option that avoids this by using the actual data distribution in tree nodes rather than the marginal distribution.

LIME is locally unstable. Multiple LIME runs on the same instance can produce different feature orderings. For high-stakes explanations, use num_samples=1000 or higher to reduce variance, or prefer SHAP.

Both methods explain correlations, not causality. If income and credit score are correlated, SHAP may attribute the prediction effect to one or the other depending on model structure, even if removing either would change the output by the same amount. Causal attribution requires intervention-based methods (do-calculus), not just prediction-based ones.

Kernel SHAP is slow for high-dimensional data. With 100+ features and a neural network, KernelSHAP becomes computationally expensive. For tabular neural networks with many features, consider training a surrogate tree model and running TreeSHAP on that surrogate.

Choosing Your Explainability Method

Click to expandDecision tree for choosing the right XAI method based on model type, scope, and regulatory context

The decision comes down to three questions: What type of model are you explaining? Do you need global or local scope? How much speed matters?

For tree-based models in production — the most common case for tabular data — TreeSHAP is the answer in almost all situations. It's exact, fast, and the only method that gives you consistent global and local explanations from the same framework.

For neural networks serving real-time predictions, LIME is often the practical choice despite its instability. DeepSHAP works but requires careful setup. KernelSHAP is a last resort for models where nothing else is available.

The main case to avoid SHAP is when you genuinely need only a quick prototype explanation during exploration, not for any artifact that will be reviewed or reported. LIME's simplicity makes it good for quick sanity checks.

When the primary goal is actionable advice to the end user — "here's what you can change" — add DiCE counterfactuals on top of SHAP attributions. They answer complementary questions.

Conclusion

SHAP and LIME are mature, production-ready tools that belong in every ML engineer's workflow, not just the model audit. SHAP 0.51.0's Shapley values offer the most theoretically grounded explanation method available — efficient, symmetric, and consistent by construction. TreeSHAP and FastTreeSHAP v2 make this practical even for large-scale inference. LIME's strength remains flexibility and speed: any model, any framework, fast enough for real-time use when approximate answers suffice.

The EU AI Act's August 2026 compliance deadline has pushed explainability from optional analysis to required infrastructure. Storing SHAP values alongside predictions, generating SHAP beeswarm plots for global documentation, and logging DiCE counterfactuals for adverse decisions now belong in your model deployment pipeline alongside accuracy metrics and latency dashboards.

For the foundational ML skills that make these tools more powerful, the articles on gradient boosting from scratch and ensemble methods are solid starting points for understanding the tree-based models where SHAP shines most. If your work involves production ML infrastructure, the MLOps guide covers how to integrate SHAP monitoring into a full deployment pipeline.

Start with TreeSHAP on your next tree-based model. Generate the beeswarm plot before you write the model evaluation report. The pattern of explanations will almost certainly show you something your accuracy metrics missed.

Interview Questions

What is the difference between global and local model interpretability?

Global interpretability describes a model's behavior across all predictions — which features matter overall and how they generally affect output. Local interpretability explains a single specific prediction — why did this particular applicant get denied? SHAP provides both: aggregate mean absolute SHAP values for global importance, and per-instance waterfall plots for local explanations. LIME provides only local explanations.

How does LIME generate explanations for a tabular model?

LIME perturbs the input instance by sampling nearby points, passes all perturbed samples through the black-box model to collect predictions, weights each sample by its proximity to the original instance using an exponential kernel, and fits a weighted linear regression on the perturbed data. The linear model's coefficients become the explanation. Because sampling is stochastic, running LIME twice may yield different results — this is the method's main weakness.

Explain the Shapley value axioms and why they matter for model explanations.

Shapley values satisfy three axioms: efficiency (all values sum to the prediction minus base value), symmetry (features with identical marginal contributions get identical values), and the dummy axiom (features that never change predictions get zero value). These axioms guarantee that SHAP explanations are not arbitrary — they are the unique attribution method satisfying all three. This makes SHAP defensible in regulatory contexts where explainability methods are scrutinized.

What is TreeSHAP and why is it faster than KernelSHAP?

KernelSHAP computes Shapley values by sampling feature subsets and approximating the combinatorial sum. TreeSHAP exploits the tree structure directly — each prediction corresponds to a path through the tree, and the algorithm propagates Shapley values along that path rather than enumerating subsets. The result is exact (not approximate) Shapley values in polynomial time, versus sampling-based approximation for KernelSHAP. FastTreeSHAP v2 pushes this further, running 2.5x faster than the original TreeSHAP algorithm.

What are SHAP interaction values, and when would you use them over regular SHAP values?

SHAP interaction values decompose each prediction into a matrix of pairwise feature effects, where the off-diagonal entry (i, j) captures how much features $i$ and $j$ jointly influence the prediction beyond their individual main effects. Regular SHAP values aggregate these interactions into a single number per feature. You'd use interaction values when you suspect features have conditional effects — for example, when credit score matters more for certain income brackets — or when you need to document model interactions for regulatory compliance. They're computationally heavier and most useful for model auditing, not real-time explanations.

How do counterfactual explanations (DiCE) differ from SHAP explanations, and when would you use each?

SHAP explains what drove the current prediction — "credit score contributed +0.21 to your approval probability." DiCE explains what would need to change — "if your debt ratio were 0.28 instead of 0.40, you would have been approved." SHAP is better for model debugging, feature selection, and documenting global model behavior. DiCE is better for communicating actionable paths to end users and for meeting the "right to explanation" spirit of regulations like the EU AI Act, which require explanations that allow affected persons to understand and contest decisions.

Your SHAP beeswarm plot shows loan_amount has low global importance but individual force plots sometimes show large loan_amount contributions. How do you explain this?

Low global importance (low mean absolute SHAP value) and high local importance for specific instances are completely compatible. Mean absolute SHAP averages effects across all instances. If loan amount only matters for a small subset of applicants — say, those at the edge of the debt ratio threshold — its average effect will be small even though it's decisive for those edge cases. This conditional behavior is exactly what SHAP interaction values would reveal: large loan_amount × debt_ratio interaction entries for instances near the decision boundary.

What are the main limitations of SHAP for real-world deployment?

SHAP's main limitation is the feature independence assumption: replacing absent features with marginal expectations produces unrealistic counterfactuals when features are correlated. The tree_path_dependent option in TreeSHAP partially addresses this by using actual data distribution in tree nodes. A second limitation: SHAP values measure correlational influence on predictions, not causal effect — if your model learned a spurious correlation, SHAP will faithfully report that spurious relationship. For high-stakes decisions, SHAP explanations should be validated by domain experts, not treated as ground truth.