ModelMRI
A local-first debugging tool that visualizes what's happening inside an AI model — attention, features, and agent steps — without cloud infrastructure.
🔗 Visit ModelMRI
Description
Understanding why an AI model produced a particular output is usually a black box — you can see the input and the output, but not what happened in between. ModelMRI opens that up: it's a local-first tool for inspecting the internals of language models, vision-language models, and even robot control policies while they run, visualizing which attention heads drove a prediction, exploring learned features through sparse autoencoders, and tracing the step-by-step execution of an AI agent.
It supports ablation analysis (turning off specific attention heads to see what changes), activation patching to pinpoint exactly where a decision is made, and can analyze both standard transformer architectures and custom nn.Module models. Results can be saved as shareable .mri files viewable without a GPU, and it includes a malicious-payload scanner for model checkpoints — a real concern given how often pretrained weights get downloaded from untrusted sources. It's free and open source under the MIT license, and runs entirely locally with no cloud dependency.
💬 Our review
The short version: a serious interpretability tool for ML researchers and engineers who need to actually see inside a model rather than treat it as a black box — the breadth (language models, VLMs, robot policies, and AI agent traces all in one tool) is unusual.
It sits in a field with several established interpretability libraries — TransformerLens, BertViz, nnsight, Neuronpedia, SAELens — which ModelMRI itself references as related tools; its distinguishing angle is bundling attention visualization, SAE feature exploration, and agent-execution tracing together in one local-first package rather than requiring separate tools for each. For debugging AI agent behavior specifically (as opposed to pure model interpretability), it also overlaps with observability tools like Langfuse, Phoenix, and LangSmith, though those are typically cloud/production-monitoring focused rather than internals-level debugging. Being free, open source, local-first, and GPU-optional for viewing shared results makes it accessible for researchers without major infrastructure.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Open source sous licence MIT, fonctionne entièrement en local.
Pros
Couvre modèles de langage, VLM et politiques robotiques dans un seul outil
Visualisation d'attention, exploration de features SAE et traçage d'agent combinés
Fichiers d'analyse partageables visualisables sans GPU
Gratuit, open source (MIT), 100% local, sans dépendance cloud
Cons
Ensemble de fonctionnalités large, courbe d'apprentissage réelle
Public de niche : chercheurs/ingénieurs ML
Concurrence plusieurs bibliothèques d'interprétabilité établies et spécialisées