AI Diplomacy
Every · Other
AI Diplomacy is Every's open-source LLM game benchmark where seven frontier AI models play the board game Diplomacy and fight to dominate Europe, with alliances and betrayals in the mix.
About AI Diplomacy
What Is AI Diplomacy
AI Diplomacy is a benchmark built by Every that turns the classic strategy board game Diplomacy into a test of how well large language models negotiate, plan and deceive. Seven AI models take the seats of seven European powers in 1901, then scheme against one another across a full game to control the map. Every built it to answer a question ordinary tests can't: which models can you actually trust when the stakes are high and cooperation and betrayal are both valid moves.
The setup plays to something Diplomacy is famous for. In the real board game, nobody wins alone. You have to talk other players into helping you, and the rules never force them to keep their word. That mix of negotiation and backstabbing is exactly what makes it hard for AI, because a model that can't read intent or hold a convincing conversation gets boxed out early. Watching the games, Every found sharp differences between models: OpenAI's o3 ran long deceptions, Anthropic's Claude models kept choosing peace even when it cost them the game, and DeepSeek's R1 leaned hard into dramatic threats.
It's a research benchmark and a spectator experience at heart, not a product you sign up for. The code is open-source on GitHub, and the games ran as a live Twitch stream, so the main way to use this LLM game benchmark is to watch, study the runs or run the code yourself. It measures how models behave under pressure, and that behavior is still a narrow slice of what they can do.
Getting Started
- Open the AI Diplomacy page at every.to/diplomacy to watch recorded games and read the results write-up
- Tune in to the Twitch stream to watch models negotiate and betray each other turn by turn
- Read the companion piece on how the project was built to see the design choices behind the benchmark
- Pull the open-source code from GitHub to run your own games, swap in models and test them on your own terms
Product Information
A quick look at AI Diplomacy's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- AI researchers and evaluators
- People who follow AI closely
- AI product teams choosing a model
- Developers who want to tinker
Tasks
- Comparing models on negotiation
- Testing deception and trust
- Stress-testing long-horizon planning
- Generating training data
- Benchmarking under new rules
Scenarios
- You need a benchmark that ordinary people will actually watch
- A model aces your knowledge tests but you don't know how it behaves when it has to cooperate
- You want to show a non-technical audience what makes today's models different
- You're deciding which model to trust for an agent that plans over many steps
Key features
Seven Models, One Game
Each match seats seven frontier language models as the great powers of 1901 Europe, and they compete until one dominates the map. Running a full game against six live opponents is a different kind of test than answering questions in isolation, and it's the core of what AI Diplomacy measures.
Negotiation and Private Diaries
Models spend each round sending messages, some public and some private, to talk rivals into cooperating. Every one of them keeps a private diary with its goals and its read on the other players. That record shows what a model said and what it was thinking at the same time.
Relationship Tracking
Models score each opponent on a scale from enemy to ally and update those scores as the game goes on. That running record is what lets them decide who to work with and who to betray, and it gives observers a clear read on how each model builds and burns trust.
Deception and Betrayal Analysis
The system compares what a model promised with what it did. It then flags betrayals and lies and sorts them into intentional and accidental. For anyone curious about how honest models are under pressure, this is the feature that answers the question directly.
Interactive 3D Viewer
Every built a browser-based 3D interface with the Three.js library. It replays a game with unit movements, battle animations and the running chat, so a dense strategy game stays followable at a glance. That ease of viewing is a big part of why the benchmark reached a wide audience.
Open-Source Codebase
The whole project is open on GitHub, with support for OpenAI, Anthropic, Google, DeepSeek and OpenRouter models. You can run your own games, swap the model lineup, adjust the rules and reuse the logs, which turns the benchmark into something you can build on rather than just watch.
Pros and cons
Pros
- Tests models on negotiation, trust and long-term planning, the social side of AI that knowledge benchmarks ignore
- Open-source code lets you run your own matches and swap in any models you want
- Runs produce rich data, including messages, private diaries and order logs, that can be reused for research or training
- The 3D viewer and live stream make model behavior understandable to people without a technical background
- The rules can be changed, so you can probe questions like how honesty affects who wins
Cons
- It's a research benchmark with no sign-up product, so you can't just log in and play a game yourself
- Watching or running games takes time, since a single match can stretch from one to dozens of hours
- Results depend on the model mix, the rules and the prompts, so comparisons only hold when you keep the setup fixed
- Test games can get expensive to run at scale, since each match calls several frontier models many times
Frequently asked questions
It's a benchmark from Every where seven frontier AI models play the board game Diplomacy against each other. The models negotiate, form alliances and betray one another to control a map of 1901 Europe, and Every uses the games to compare how they handle trust and strategy.
Related content
Explore related tools, skills, and articles for AI Diplomacy.
AI Diplomacy Alternatives
JarvisCore
Prescott Data · Coding · Productivity · OtherJarvisCore is an open source Python runtime where AI agents work as equal peers on an agent mesh, claim tasks from a shared ledger, and reach external services through a zero-trust broker.
Clef
Cloudflare · Business · OtherClef is Cloudflare's open-source decision model for Workers AI. It returns calibrated probabilities for typed questions, not generated text, so agents can classify, route, and act fast.
iFixAi
iFixAi · Business · OtheriFixAi is an open-source auditor for AI agents. It runs your agent through a simulation of your business, grades every reply, and returns an A to F scorecard backed by evidence.