A murder mystery tells you John handed Mary the knife, then mentions that the knife sat in a backpack two chapters earlier. AI2 built a benchmark around whether a model keeps up with that kind of change.
What StateBench measures in a long document
AI2 published StateBench on November 3, along with a technical report, the data and the code. It tests state tracking, which means whether a model keeps up as facts about people and things change across a long text.
That's different from the usual long-context test. Retrieval asks whether a model can find one sentence buried deep in the text. State tracking asks whether it knows who holds the knife after a few handoffs.
The team's framing is a mystery novel where John gives Mary the knife just before a murder. A reader updates who has it. A harder version: John hands over his backpack, and the knife was placed inside it earlier by Edward. Now the answer depends on two updates, not one.
The four StateBench levels, from trivial to general finite
The benchmark splits tasks by the kind of assignment involved, and each level gets a short label from programming.
r-trivial covers values that never change, like a fixed value of 4 or a birth date. aperiodic covers reassignment by value, like a thermostat going from 4 to 2. soluble covers self-reference, like a running score that adds five points. general finite covers assignment by reference, the swap case where one variable takes another's value and two of them trade places.
The shell game with five cups is the everyday version of that last one. All four levels use the same number of variables, values and update positions, so a score drop between them reflects the kind of assignment and nothing else.
How Olmo 3 and Olmo Hybrid handled state tracking
The pretrained models were Olmo 3, a standard transformer, and Olmo Hybrid, which adds Gated DeltaNet layers to the mix, plus 275M and 2.7B versions of each. All of them trained on the same data, and none saw contexts beyond 64,000 tokens during training, yet AI2 tested them well past that limit to see what broke first.
Almost all of them handled r-trivial. Accuracy fell on aperiodic as inputs grew longer. On general finite, they sat at chance, which means the models were guessing.
Chance is the important result there. A model that guesses on the hardest level isn't degrading gracefully. It has no handle on the problem at all.
Why format changed the AI2 benchmark results
The most uncomfortable finding is about presentation. The 275M hybrid and Gated DeltaNet models scored near the ceiling on r-trivial problems written in the RULER style. Reframed as spreadsheet edits, the same problems dropped them to near chance.
Same logic, same answer, different wrapper. That's why StateBench tests several formats instead of one, and it's a warning for anyone reading a single benchmark score as a capability claim.
Frontier models on the long context benchmark
Four frontier models went through the long context benchmark: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna.
All four did well on r-trivial and aperiodic. On soluble and general finite, accuracy fell sharply as the documents got longer, dropping to 13% or lower on the longest general finite problems the team ran.
What does that look like in a real document? AI2 points to sensitivity as the explanation. At that depth, a single wrong step changes the answer, and small errors compound across a long chain of updates.
What the from-scratch StateBench runs showed
The team then trained six 275M models from scratch: two transformers, two hybrids and two Gated DeltaNet variants. Training used sequences of up to 256 assignments, and testing went to 64,000, which is 250 times the training length and far beyond anything the models had seen.
Gated DeltaNet and the hybrid models generalized almost perfectly on the three easier levels. The transformers degraded on long sequences and never learned general finite at all.
The most flexible Gated DeltaNet stayed near perfect out to about ten times its longest training sequence. That's the closest thing to good news in the paper, and it's about architecture rather than size.
The Gated DeltaNet proof, and what it doesn't settle
There's a formal result buried in the report. A Gated DeltaNet with a non-negative eigenvalue memory update can solve all aperiodic tasks and some periodic ones under fixed-precision arithmetic.
That's a proof about a model family, not about the ones AI2 trained. Where exactly this sits in the hierarchy of state-tracking problems remains open, and so does the question of whether it solves all soluble tasks.
What StateBench still can't test
The stated limitation is a fixed number of variables. Real conversations introduce new people, new objects and new names as they go, and the current design doesn't model that.
So treat the results as a floor rather than a verdict on long context generally, since the benchmark only asks about a cast that never changes size. A model that struggles when the cast is fixed will struggle more when someone new walks into the story on page 40.
