Captum · Model Interpretability for PyTorch
Meta (PyTorch team) · Coding
Captum is a free, open source library from Meta that explains PyTorch model predictions. Developers use its attribution algorithms to see which input features, neurons or layers drove an output.
About Captum · Model Interpretability for PyTorch
What Is Captum
Captum is a Python library for PyTorch model interpretability, built to show what a trained network is actually reacting to. The name is Latin for comprehension, and that's the job: take a model you already built, point an attribution algorithm at it, and get back numbers that say how much each input feature, neuron or layer contributed to a specific prediction.
It matters most when a model misbehaves and nobody can say why. A classifier flags a scan as abnormal, a recommender keeps pushing the same odd item, a language model answers a question with a strange confidence. Captum gives you a way to look inside the forward pass instead of guessing.
The catch is scope. Captum is a library, not an app. You need working Python and PyTorch code before any of it makes sense, and the results are tensors you have to interpret yourself or plot in a notebook. There's no hosted dashboard to open. The team retired the Captum Insights visual interface after v0.8.0, so anyone expecting a point and click tool will be disappointed.
Getting Started
- Install the package with
pip install captum, orconda install captum -c pytorchif you use conda. Version 0.9.0 needs Python 3.10 or newer and PyTorch 2.3 or newer. - Load your trained model and switch it to eval mode. No architectural changes are needed for most algorithms.
- Pick an algorithm and pass it your model.
IntegratedGradients(model)is the usual first choice for gradient based attribution. - Call
attribute()with your input batch and, where the algorithm asks for one, a baseline tensor such as all zeros. - Inspect the returned attributions, compare the convergence delta if you requested it, and plot the scores next to your inputs.
Product Information
A quick look at Captum · Model Interpretability for PyTorch's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- Machine learning engineers
- Interpretability researchers
- Application engineers running models in production
Tasks
- Debugging a wrong prediction
- Checking whether a model learned shortcuts
- Comparing neurons and layers
- Auditing training data
Scenarios
- Preparing a model for a review meeting
- Research on large language models
- Testing resilience before release
Key features
Attribution Algorithms
Captum groups its methods into gradient based and perturbation based families. Gradient methods include Saliency, InputXGradient, IntegratedGradients, DeepLift, GradientShap, GuidedBackprop and Guided GradCam, while perturbation methods include FeatureAblation, Occlusion, ShapleyValueSampling and Lime. Having both families in one place means you can cross check a result: when a gradient method and a perturbation method point at the same pixels, the feature attribution is more likely to hold, so you can bring a specific explanation to a model debugging session instead of a hunch.
Neuron and Layer Attribution
Feature attribution tells you what the model looked at. Neuron and layer attribution tells you where inside the network that happened. LayerConductance, LayerIntegratedGradients, LayerLRP and their neuron counterparts score activations at a chosen layer, which is how you trace a broken prediction back to a specific point in the architecture.
LLM Attribution Wrappers
LLMAttribution and LLMGradientAttribution wrap the existing algorithms so they work on token sequences from Hugging Face style language models. They report which prompt tokens influenced a generated answer, and you can group tokens into words or phrases before scoring them. LLM attribution is slower than a single forward pass, so plan around inference cost.
Concept and Influence Analysis
The concept module implements TCAV, which tests whether a concept like stripes or fur is important to a class by training a small classifier on activations. The influence module adds TracIn and similar estimators for asking which training examples shaped a prediction. Both are aimed at research workflows rather than quick checks.
Robustness Testing
The robustness module provides FGSM, PGD, AttackComparator and MinParamPerturbation. You can generate adversarial examples, compare attacks side by side and search for the smallest perturbation that flips a label. This pairs naturally with attribution, since a feature that survives both a gradient check and an attack check is a stronger finding.
Metrics for Explanation Quality
Infidelity and Sensitivity metrics estimate how trustworthy a set of attributions is. Instead of trusting a heatmap because it looks reasonable, you can quantify how much the explanation disagrees with the model when inputs are perturbed, and compare algorithms on the same sample.
Pros and cons
Pros
- No model rewiring: most algorithms accept any PyTorch `nn.Module` as is, so you don't have to retrain or restructure a network to explain it.
- Wide algorithm coverage: more than sixty exported attribution classes, from Saliency to LRP, Shapley values and TCAV, all under one import path.
- Verifiable results: IntegratedGradients returns a convergence delta and Captum ships infidelity and sensitivity metrics, so explanations can be checked rather than assumed.
- Multimodal by design: the same API handles vision tensors, text token inputs, audio and time series, which keeps one code style across projects.
- Free with a permissive license: BSD 3 Clause means you can use it in commercial code without asking anyone.
Cons
- Steep learning curve: choosing between forty odd algorithms requires understanding what each one assumes, and the library doesn't help you decide.
- No visual interface: Captum Insights was retired after v0.8.0, so you have to write your own plots to see anything.
- Compute heavy on some methods: perturbation based algorithms and LLM attribution run many forward passes per sample, which gets expensive on large models.
- Requires PyTorch knowledge: there's no framework agnostic wrapper for people outside the PyTorch world, and the outputs arrive as raw tensors. Teams on TensorFlow or JAX can't reuse their existing model code, and anyone without Python skills has no entry point.
Frequently asked questions
Captum explains predictions from PyTorch models. It scores how much each input feature, neuron or layer contributed to an output, which helps with debugging, model review and research on how networks make decisions.
Related content
Explore related tools, skills, and articles for Captum · Model Interpretability for PyTorch.
Captum · Model Interpretability for PyTorch Alternatives
Base Code
Base44 (Wix) · Coding · ProductivityBase Code is a shared cloud development environment from Base44. Teammates describe changes to an existing GitHub codebase in chat, preview them live and send them back as pull requests.
Google EmbeddingGemma 2
Google DeepMind · Writing · Image · CodingGoogle EmbeddingGemma 2 is a 740M-parameter open model that turns text, code, images, audio, and video frames into vectors in one 768-dimension space, running on phones and laptops.
GitGlow
GitGlow · Coding · ProductivityGitGlow is a local-first desktop Git client that lets independent developers review their own code and their AI agents' changes in one place before shipping.