Explaining Seldon Predictions with Shapley Values

Neuralk’s SeldonClassifier and SeldonRegressor are black-box foundation models, but you can still understand why they make a given prediction. The neuralk.explainability module wraps the shapiq package to provide:

• local explanations — how each feature (and feature interaction) contributed to a single prediction, and

• global explanations — overall feature importance aggregated across many predictions.

Note

Explanations require the optional dependency, installed with pip install "neuralk[explain]". It targets Python 3.12+.

For this example to run, the environment variable API_KEY must be set with your Neuralk API key.

Warning

Seldon predicts through remote API calls that consume credits. Explainers evaluate the model on many perturbed inputs, so keep the background set small and the estimation budget modest — as we do below.

Fit a classifier

We use a small, synthetic dataset where only the first two features carry signal, so the explanations have a clear ground truth to recover.

import os

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

from neuralk import SeldonClassifier
from neuralk.explainability import SeldonExplainer

X, y = make_classification(
    n_samples=80,
    n_features=5,
    n_informative=2,
    n_redundant=0,
    random_state=0,
)
feature_names = [f"feature_{i}" for i in range(X.shape[1])]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0
)

# Assigned, not left as the cell's last expression: the gallery would render
# the estimator there, API key included.
clf = SeldonClassifier(api_key=os.environ.get("API_KEY")).fit(X_train, y_train)

Build an explainer

SeldonExplainer wraps the fitted classifier. The background data is used to fill in “absent” features while estimating Shapley values; a subset of the training data works well. We request max_order=2 so pairwise feature interactions are estimated too.

explainer = SeldonExplainer(
    clf,
    background=X_train,
    feature_names=feature_names,
    index="k-SII",
    max_order=2,
    max_background_samples=40,
    random_state=0,
)

Local explanation

Explain a single prediction. budget controls how many model evaluations shapiq may use — higher is more accurate but slower (and costs more credits).

local = explainer.explain(X_test[0], budget=128)
print(local)

/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
InteractionValues(
    index=k-SII, max_order=2, min_order=0, estimated=False, estimation_budget=128,
    n_players=5, baseline_value=0.4850693643093109,
    Top 10 interactions:
        (): 0.4850693643093109
        (1, 4): 0.04725679230227252
        (3, 4): 0.0149738462320378
        (2,): 0.013976146339672018
        (3,): -0.0022725993758701294
        (1, 2): -0.002481457044983662
        (1, 3): -0.00958788087736256
        (2, 4): -0.011287395375390431
        (1,): -0.04681731855873862
        (4,): -0.46650091549989325
)

Visualize the contribution of each feature with a force plot.

explainer.plot_force(local, show=True)

• 
• 

Global explanation

Aggregate explanations over several test instances to see which features matter on average. explain_global runs one local explanation per instance, so its cost grows with the number of rows: we explain a modest subset.

global_explanation = explainer.explain_global(X_test[:10], budget=128)

/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)
/home/runner/work/neuralk/neuralk/.venv/lib/python3.12/site-packages/shapiq/approximator/regression/base.py:168: UserWarning: Not all budget is required due to the border-trick.
  self._sampler.sample(budget)

Rank features by mean absolute contribution:

importances = global_explanation.feature_importances(normalize=True)
for name, score in importances.items():
    print(f"{name:>12}: {score:.3f}")

feature_4: 0.896
feature_1: 0.048
feature_2: 0.037
feature_0: 0.010
feature_3: 0.008

And the corresponding aggregated bar plot:

global_explanation.plot_bar(show=True)