Magma
Microsoft Research · Other
Magma is Microsoft's foundation model for multimodal AI agents. It reads images, video and text, then plans actions like clicking a button or moving a robot arm in digital and physical worlds.
About Magma
What Is Magma
Magma is a foundation model built by Microsoft Research for multimodal AI agents. Unlike a standard vision-language model that only describes what it sees, Magma pairs understanding with action. It takes in images, video and text, then proposes goal-driven steps, from clicking buttons in a mobile app to directing a robotic arm to pick up an object. The paper was accepted at CVPR 2025, and Microsoft released the code and model weights openly.
The idea behind it is one model for two worlds. Most earlier systems were trained for a single job, either 2D interfaces or 3D robots, and struggled to move between them. Magma was pretrained on a mix of images, video and robotics data so a single model can handle UI navigation, robot manipulation and general image or video questions. Microsoft calls it the first foundation model for multimodal AI agents.
The main limit is that Magma is a research model, not a finished product. It ships as open weights and code you run yourself, and you'll need a capable GPU to try it. It's built for developers and researchers who can finetune it, not for casual users who want a ready-made app.
Getting Started
- Set up a Python environment with PyTorch and the Magma repo from GitHub.
- Download the Magma-8B weights from Hugging Face.
- Load the model and processor, pointing the code at your GPU.
- Feed it an image or video plus a text prompt to get an answer or a proposed action.
- Finetune on your own data for a specific task such as UI navigation or robot control.
Product Information
A quick look at Magma's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- AI researchers
- App and robotics developers
- Students learning multimodal AI
Tasks
- UI navigation
- Robot manipulation
- Image and video questions
- Spatial reasoning
Scenarios
- Research on agentic models
- Building an interface automation prototype
- Exploring cross-domain transfer
Key features
One Model for Digital and Physical Tasks
Magma handles both UI navigation and robot manipulation in a single set of weights. Earlier models were usually tuned for one environment, so a screen agent couldn't drive a robot. Microsoft trained Magma on mixed data so it can switch between clicking an app and planning a physical action.
Set-of-Mark for Action Grounding
Set-of-Mark labels clickable objects in an image, such as buttons on a screen or a gripper on a robot arm, with numeric marks. The model predicts which mark to act on, which turns loose visual understanding into a concrete next action.
Trace-of-Mark for Action Planning
Trace-of-Mark labels how objects move across video frames, like the path of a hand or robotic arm. By learning from these traces, Magma plans multi-step actions and predicts what should happen next instead of reacting frame by frame.
Pretraining on Mixed Data
Magma was pretrained on a large, varied set of images, video and robotics data. That mix lets one model carry general image and video understanding alongside agentic skills, rather than trading one away for the other. It leans on freely available video to learn spatial and temporal patterns.
Video and Image Understanding
Beyond actions, Magma answers questions about photos and clips, including video QA and spatial reasoning tasks. Microsoft reports it competes with larger vision-language models on several benchmarks despite training on less video instruction data.
Open Code and Weights
Microsoft publishes the model weights on Hugging Face, the training and inference code on GitHub, and makes Magma-8B available on Azure AI Foundry. That openness lets researchers reproduce results and finetune the model, but it leaves setup and hosting to you.
Pros and cons
Pros
- One model spans UI navigation, robot control and general image and video understanding.
- Open weights and code let researchers reproduce results and finetune freely.
- Set-of-Mark and Trace-of-Mark turn visual reading into concrete action plans.
- Reports state-of-the-art results on UI navigation and robot manipulation tasks.
Cons
- It's a research model, so there's no finished app and setup takes real work.
- Running Magma needs a capable GPU, which leaves casual users out.
- Results still depend on finetuning for a specific task, so out-of-the-box use is limited.
- Some data and evals come from Microsoft itself, so independent checks matter.
Frequently asked questions
Magma is a foundation model for multimodal AI agents. Developers use it to build agents that read a screen and propose clicks, or that plan robot movements, along with general image and video understanding. It's research code, so you finetune it for your own task.
Related content
Explore related tools, skills, and articles for Magma.
Magma Alternatives
Odyssey 3
Odyssey · OtherOdyssey-3 is Odyssey's foundation world model. It simulates physics in real time, turns a prompt into an interactive environment, and adapts to robots, cars and games.
JarvisCore
Prescott Data · Coding · Productivity · OtherJarvisCore is an open source Python runtime where AI agents work as equal peers on an agent mesh, claim tasks from a shared ledger, and reach external services through a zero-trust broker.
Clef
Cloudflare · Business · OtherClef is Cloudflare's open-source decision model for Workers AI. It returns calibrated probabilities for typed questions, not generated text, so agents can classify, route, and act fast.