A model scores 94% on a held-out test set and a team ships it. Six weeks later it fails on a category of inputs nobody tested, and the postmortem question is always the same: what was it actually doing? Not what it was trained to do, not what the loss function rewarded — what computation produced a wrong answer delivered with high confidence. Test-set accuracy can’t answer that. It confirms the model performs well on the distribution it was measured against, and says nothing about the mechanism behind that performance — which is exactly what you need once the distribution shifts, a regulator asks for an explanation, or the failure mode matters more than the aggregate score.
Interpretability research exists to close that gap. It’s not one method but a loose federation of techniques — some post-hoc and statistical, some structural and mechanistic — aimed at variants of the same question: what did this network learn, and how does it use that in producing a given output. This piece surveys the major approaches in current use, is specific about what each one measures, and equally specific about where each runs out of explanatory power. Lab affiliations are noted where relevant to a technique’s origin, not as endorsement of any one research program’s framing of the problem.
Why Interpretability Has Moved From Nice-to-Have to Load-Bearing
Three pressures have pushed interpretability from an academic sideline into something teams budget for directly.
Debugging. When a model misclassifies, the practical question is whether the error is a data problem, a labeling problem, or a genuine gap in what the model learned. Attribution methods that surface which input features drove a decision let engineers distinguish “the model latched onto a spurious correlation” from “this input is genuinely ambiguous.” The canonical example is a model that learned to associate snow with wolves rather than the animal itself, because every wolf photo in training happened to include snow — invisible in aggregate accuracy, immediately visible in a saliency map. A related but distinct failure category — a model that degrades on inputs exceeding its effective working memory rather than its labeled concepts — is covered in our piece on long-context retrieval limits; the debugging instinct is the same even though the underlying mechanism differs.
Safety. As models are deployed in settings with real consequences — medical triage support, credit decisions, autonomous systems — the cost of an undetected failure mode rises. Interpretability is one of a small number of tools that can catch a problem before deployment, by revealing whether a model’s internal computation tracks the concept it’s supposed to track or a proxy that merely correlates with it during training. This concern is sharpest for behaviors instilled indirectly through reinforcement learning from human feedback, where a model is optimized against a learned reward signal rather than explicit labels, and the distance between the intended objective and what the reward actually rewards is exactly the kind of gap that outputs alone rarely reveal.
Trust and regulatory pressure. Teams deploying models into regulated domains increasingly face a documentation requirement, not just a technical one. The EU’s regulatory framework for high-risk AI systems includes transparency and explainability obligations, and U.S. sector-specific rules (fair lending, for instance) have long required adverse decisions be explicable to the person affected. Interpretability tooling is what a team points to — with the caveat, discussed below, that current tools vary in reliability, and treating a saliency map as a legal-grade causal account is a mistake auditors catch.
None of these pressures are satisfied by test-set performance alone — the practical case for methods most ML engineers meet piecemeal, a SHAP call here, a probing classifier there, without a map of how the field organizes itself.
Feature Attribution: Saliency Maps, Integrated Gradients, and SHAP
The most widely deployed family of interpretability methods answers a narrower question than “what did the model learn”: given one input and one output, which parts of the input mattered most to that prediction? These are local, post-hoc explanations — they don’t touch the model’s weights or generalize across inputs. They’re attached after training, to a specific decision.
Saliency maps are the oldest and simplest of these. The technique computes the gradient of the output with respect to the input — for an image classifier, the gradient of the predicted class score with respect to each pixel — and visualizes the magnitude as a heatmap. A large gradient at a pixel means a small perturbation there would move the prediction more than elsewhere. The appeal is that it requires nothing beyond a single backward pass. The known weakness is that raw gradients are noisy and can saturate: past a certain point, increasing a feature’s value stops changing the gradient even though the feature is clearly relevant, so vanilla saliency maps understate features the model already relies on heavily.
Integrated Gradients, introduced by Sundararajan, Taly, and Yan in “Axiomatic Attribution for Deep Networks”, addresses the saturation problem directly. Instead of a single gradient at the actual input, it integrates gradients along a straight-line path from a baseline (often an all-zeros input) to the actual input, summing how much each feature’s contribution accumulates across that path. It’s grounded in two axioms the authors argue any attribution method should satisfy — sensitivity (a feature that changes the output must receive nonzero attribution) and implementation invariance (identical networks should produce identical attributions) — and the paper shows several earlier methods violate one or both. It’s costlier than a single backward pass (attributions approximate a Riemann sum, typically 20–300 steps) but has become a standard baseline.
SHAP (SHapley Additive exPlanations), from Lundberg and Lee’s “A Unified Approach to Interpreting Model Predictions”, takes a different route. It borrows the Shapley value from cooperative game theory: treat each feature as a “player” in a game whose “payout” is the prediction, and compute each feature’s average marginal contribution across all possible orderings in which features could be added. This guarantees properties — efficiency (attributions sum to the gap between prediction and baseline), consistency, and a unique solution satisfying them — that ad hoc schemes don’t. The tradeoff is cost: exact Shapley values require an exponential number of feature-subset evaluations, so practical implementations (KernelSHAP, TreeSHAP, DeepSHAP) use sampling or model-specific shortcuts. SHAP has become close to a default for tabular-data explanation in industry because its output — a signed contribution per feature, comparable across a dataset — maps onto how stakeholders already think about “what drove this outcome.”
All three share a structural limit worth stating plainly: they explain one prediction relative to a baseline. They don’t tell you what the model learned in general, only how the current output would change if the current input changed — a real answer, but a narrower one than “understand the model.”
Probing Classifiers: What Do the Representations Encode?
A different family of methods asks about the model’s internal representations rather than the input-output mapping directly: does a given hidden layer encode a specific piece of information, whether or not the output task requires it?
The probing classifier approach is mechanically simple. Freeze the pretrained model, extract activations from a layer of interest, and train a small auxiliary classifier — usually linear or shallow — to predict some property (part of speech, syntactic depth, sentiment, factual attributes) from those frozen activations. High probe accuracy is standardly read as evidence the layer “encodes” that property in a linearly (or near-linearly) accessible way.
Probing has produced genuinely informative results about language models — for instance, work showing that BERT’s intermediate layers encode a rough approximation of the classic NLP pipeline (part-of-speech tagging earlier, syntactic and semantic structure later), suggesting the network reconstructs something like traditional linguistic structure without being explicitly trained to. But the method has a well-documented trap: probe accuracy conflates “the information is present in the representation” with “the probe is powerful enough to extract it regardless of whether the original model uses it.” A sufficiently expressive probe can learn to extract a property from activations that contain only a weak, incidental correlate of it — so high probe accuracy doesn’t guarantee the base model actually relies on that information. Controlling for this requires control tasks — probing for a property the model has no reason to encode, to establish a baseline for how much a probe of a given complexity can memorize on its own, per the methodology proposed by Hewitt and Liang. Probing without a control task is a common, easy-to-miss error, and its results should be read skeptically without one.
Mechanistic Interpretability: Circuits and Superposition
Feature attribution and probing are, by design, agnostic about how the network computes what it computes — they treat it mostly as a function queried from outside. Mechanistic interpretability takes the opposite stance: it tries to reverse-engineer the actual algorithm implemented in a network’s weights, at the level of individual neurons, attention heads, and connecting circuits — treating the trained network the way a reverse engineer treats compiled binary.
The clearest illustration is induction heads — attention head pairs, identified through direct analysis of small transformer weights, that implement a simple in-context copying algorithm: given a sequence like “…A B … A”, an induction head attends back to the prior occurrence of A and predicts B will follow again. This pattern, described in detail in interpretability work published by Anthropic’s research team as one contributor among several groups studying transformer internals, was shown to correlate with much of the in-context learning behavior observed in small transformers — a genuinely causal claim, not a correlational one, because it was derived by tracing the actual weight computation rather than observing input-output pairs.
A harder problem the same research surfaced is superposition: networks routinely represent more distinct features than they have neurons or dimensions, by encoding features as overlapping, non-orthogonal directions in activation space rather than giving each feature its own neuron. This follows from training on data where most features are sparse — active for only a small fraction of inputs — making it statistically efficient to pack more features into a space than its dimensionality would naively allow, at the cost of occasional interference between them. Superposition is a serious obstacle because it breaks the intuitive assumption of “one neuron, one concept” — a single neuron’s activation can blend several unrelated features, and no amount of single-neuron analysis separates them cleanly.
The current, still-developing response to superposition is the sparse autoencoder (SAE): train a secondary, wider autoencoder to reconstruct a layer’s activations under a sparsity penalty on its hidden layer, hoping the autoencoder’s over-complete hidden units each correspond to a single, human-interpretable feature the original network had packed into superposition. Early results across several groups suggest this can recover directions responding to specific, describable concepts rather than the polysemantic mess of a raw neuron. But SAEs remain an active research direction, not a settled tool: results depend heavily on the sparsity penalty and dictionary size chosen, there’s no fully agreed way to verify a recovered “feature” is genuinely monosemantic, and scaling to the largest production models remains unresolved.
What Interpretability Tools Do Not Give You
Two limitations run across nearly every method above, worth stating without hedging, because overclaiming interpretability results is a recurring failure mode in both research papers and vendor marketing.
Correlation with the internal state is not proof of the causal mechanism. A saliency map, a SHAP value, and a probing classifier all establish that some quantity (a gradient, a marginal contribution, a linearly-decodable signal) correlates with an input feature or a labeled property. None of them, on their own, establish that the network’s decision procedure routes through that feature causally — that intervening on it, rather than merely observing it, would change the output as predicted. Mechanistic interpretability is more disciplined here, because circuit-level claims are typically validated by ablation — removing the identified component and checking the predicted behavioral change actually occurs — but ablation-validated circuits currently cover a small fraction of what any production-scale model does. For everything else, “this feature correlates with the decision” is the honest description, not “this feature causes the decision.”
Interpretability has not scaled cleanly to the largest models. Circuit-level analysis and sparse autoencoder work have produced their clearest results on small and mid-sized models. Extending the same rigor to models with tens or hundreds of billions of parameters runs into a combinatorial problem (far more neurons, heads, and layers to characterize) and a conceptual one (no guarantee clean circuits found in small models represent how capability emerges at scale). The practical consequence: an engineer working with a frontier-scale model has meaningfully better attribution and probing tools than mechanistic ones, and a claim that a specific model’s behavior has been “fully explained” mechanistically deserves skepticism until the underlying analysis is published and reproduced.
None of this makes the tools useless — attribution reliably helps debug specific failures, and mechanistic interpretability has produced real, ablation-verified findings that inform how the field thinks about larger models. But a team building an interpretability-dependent audit should scope claims to what the specific method can actually support. For teams also working through what a model’s benchmark scores do and don’t establish, the pattern will look familiar: the number or the map is a real signal, not the whole story.
Frequently Asked Questions
What is the difference between feature attribution and mechanistic interpretability?
Feature attribution (saliency maps, Integrated Gradients, SHAP) is post-hoc and treats the model as a black box, measuring how much each input feature correlates with a given output. Mechanistic interpretability opens the box, tracing the actual weight computation — individual neurons, attention heads, circuits — to build causal, verifiable claims about the algorithm the network implements internally.
Is SHAP the same as Integrated Gradients?
No. Both are attribution methods, but they rest on different theoretical foundations. SHAP derives attributions from Shapley values in cooperative game theory, guaranteeing properties like efficiency and consistency across feature subsets. Integrated Gradients derives attributions by integrating gradients along a path from a baseline input, satisfying axioms like sensitivity and implementation invariance. They often agree in practice but aren’t interchangeable by construction.
What is superposition in neural networks?
Superposition is when a network represents more distinct features than it has neurons or dimensions, by packing multiple features into overlapping, non-orthogonal directions in activation space rather than dedicating one neuron per feature. It’s an efficient encoding strategy for sparse features but makes single-neuron analysis unreliable, since one neuron’s activation can reflect several unrelated concepts blended together.
Can interpretability tools fully explain what a large language model is doing?
Not currently. Attribution and probing methods provide reliable but narrow, correlational signals about specific predictions or representations. Mechanistic interpretability offers stronger causal claims but has been validated mainly on small and mid-sized models; extending it to frontier-scale models runs into scale and entanglement problems that remain unresolved as of mid-2024.
Why do regulators care about interpretability?
As models influence decisions in domains like lending, hiring, and medical support, regulatory frameworks increasingly require that adverse or high-stakes decisions be explicable to affected individuals. Interpretability methods are the practical tools teams use to produce those explanations, though current tools vary in reliability and should not be treated as legally definitive causal accounts of a model’s decision process.