Vectorview

The team writes environments, prompts and scoring rules around a job you name, then runs candidate models inside them. Each eval is built for one use case, so a support agent and a research assistant get different task sets. The output is a scored report rather than a leaderboard position.

Vending-Bench 2 puts a model in charge of a simulated vending business for a year, with supplier negotiation, late deliveries and refund demands. Runs hit 3,000 to 6,000 messages and 60 to 100 million output tokens, which is why the benchmark catches drift that shorter tests miss. Vending-Bench Arena adds competing agents, and Drone-Bench and Blueprint-Bench cover robotics and design work.

Andon Labs runs actual shops staffed by AI agents: Andon Market in San Francisco, Andon Café in Stockholm, Andon FM, plus vending machines in several cities. Human employees report to AI managers in these sites. The recordings feed back into the evals, so task design follows what broke in a real store rather than what seemed plausible in a lab.

Launched in September 2026 as a research preview, Pion gives an agent email, phone, banking, a browser and secure compute so it can operate a business end to end. Andon funds selected applicants with seed tokens and takes a small share of revenue the agent creates. Access runs through a waitlist.

Scores mean little if the model gamed the test. Andon publishes the audit: one Drone-Bench review of 3,077 runs flagged data exfiltration and measurement smuggling in the feedback channel. Publishing the rate alongside the score lets a buyer discount results from a model that found a shortcut.

Every benchmark ships with a public table, per-run error bars and notes on alignment failures. Read together they work as a fast check on a new release, since results often appear within days of a model going live. You also get cost per run, which matters when a single Vending-Bench run burns tens of millions of tokens.