Cekura Bench
Cekura · Voice & Language · Business
Cekura Bench is a public benchmark that runs speech-to-speech models and deployed voice agents through 82 simulated phone calls, three times each, then publishes pass rates, latency and cost.
About Cekura Bench
What Is Cekura Bench
Cekura Bench is a public leaderboard run by Cekura, a company that sells testing and monitoring tools for voice agents. It scores realtime voice models and shipped agents on simulated phone calls, using audio from start to finish instead of text transcripts.
The benchmark exists because a transcript can look perfect while the call falls apart. An agent books the right appointment in the log, then talks over the caller, mishears a date of birth, or goes silent while a tool runs. Cekura Bench puts the whole path under one score: turn detection, speech-to-text, the model, tool calls, text-to-speech and transport.
Its main limit is scope. Every scenario is an English-language call in one of two domains, appointment booking or Medicare intake, so a team with a different script will not find its exact workflow on the board. Results also arrive in batches, which means the newest model release may be missing for a few weeks.
Getting Started
- Open benchmarks.cekura.ai and choose a board: speech-to-speech models, voice agent workflows, turn-taking, speech-to-text or text-to-speech.
- Read the reliability column first. Pass³ tells you how often an agent passed all three runs of a scenario, while success rate counts single calls.
- Clone the open harness from the cekura-bench repository, point it at your agent, and run the published scenarios against it.
- Send your config to Cekura if you build a model or a platform, so it can be scored under the same rules as everyone else.
Product Information
A quick look at Cekura Bench's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- Voice AI engineers
- Contact center product teams
- Platform vendors
- Analysts and researchers
Tasks
- Choosing a speech-to-speech stack
- Running a vendor bake-off
- Checking latency budgets
- Diagnosing turn-taking bugs
Scenarios
- Before you commit to a realtime model for an inbound line, when a wrong pick is expensive to undo.
- After a provider changes pricing or ships a new model version, since cost per minute sits next to the pass rate.
- When production complaints mention interruptions, dead air or repeated questions, and you need to know if the cause is the model or the transport.
- When you have to explain a stack decision to people who will not read a call log.
Key features
Full calls in audio, not text
Each test is a multi-turn call over a real call path, so the score covers turn detection, speech-to-text, the model, tools, text-to-speech and transport. That makes it a phone call benchmark rather than a text evaluation. It matters because most production agents are cascades built on Vapi, Retell, ElevenLabs, LiveKit, Pipecat or Telnyx, and a text test never touches those layers. Cekura Bench runs against all six platforms and against realtime models directly.
A caller that follows rules, not a prompt
The simulated caller is an LLM inside a harness built on conditional actions. A test is a set of rules: when the agent asks for a date of birth, the caller answers with a fixed line; when a reply runs long, the caller cuts in at 1.2 seconds. Faults land where the test places them, including dead air, keypad tones, prerecorded audio and packet loss, so every model meets the same moment.
Pass³ reliability with confidence intervals
Each of the 82 scenarios runs three times. The board reports the pass rate and the all-three-pass rate, which it calls pass³. Every figure ships with a 95% confidence interval, and rows whose intervals overlap count as a tie. That stops small gaps from reading as wins.
Failure layers kept apart
A call passes only if every check passes, but the board breaks failures into separate measures: task completion, tool calls and arguments, interruptions, gibberish and transcription, infrastructure faults, data accuracy and response time. A model that loses points to no-connects looks different from one that saves the wrong address, which is the difference between an operator problem and a prompt problem.
Open data and an open harness
The scenario set, scripts, expected outcomes and tool calls live in a versioned dataset on Hugging Face under CC BY 4.0. The harness and reference agent sit in a public repository, and raw audio, event logs and frames are kept so a score can be recomputed. Published dataset versions are frozen, and fixes ship as a new version with a changelog entry.
Boards beyond speech-to-speech
The speech-to-speech board covers 12 realtime models plus a cascaded baseline across 3,198 calls. The voice agent benchmark covers nine platforms on 2,214 calls, 13 models on turn-taking, 20 speech-to-text models on 1,000 clips and real calls, and 26 text-to-speech models on 30,564 requests. Cost per minute uses each vendor's published rate and stays blank where no rate exists.
Governance you can audit
Cekura discloses that it sells voice AI testing to the same market it measures, and some ranked vendors are customers or partners. To offset that, configs are locked before scoring, vendors get a fixed pre-review window to flag errors, and corrections are logged in public. A neutral advisory board is being formed to settle disputes.
Pros and cons
Pros
- Audio end to end, so turn detection and transport failures count against the score instead of disappearing.
- Identical scenarios, caller behavior and interruption timing for every model, which makes a vendor bake-off worth running.
- Raw recordings and versioned datasets let a skeptical engineer recompute the numbers.
- Free to read, with results reusable under CC BY 4.0 and data downloadable as JSON.
- Failure measures are split by cause, so you can tell a model problem from a pipeline problem.
Cons
- Two domains only, appointment booking and Medicare intake, so niche workflows are untested and your own script still needs a custom suite.
- Cekura sells testing tools to the vendors it ranks, a conflict the site discloses but cannot remove.
- The expected-outcome judge prompt and some rubrics stay private, so only the tool-call score can be reproduced exactly.
- Running the suite on your own agent costs $0.25 per testing minute through the Cekura platform if you skip the self-hosted harness.
Frequently asked questions
Cekura Bench is a public benchmark for voice AI. It runs simulated phone calls against speech-to-speech models and deployed voice agents, then publishes pass rates, latency, cost per minute and failure details.
Related content
Explore related tools, skills, and articles for Cekura Bench.
Cekura Bench Alternatives
Foley
Hanamori Labs, LLC · Voice & Language · Productivity · AI AssistantFoley is a voice to-do list that turns a spoken sentence into a filed task, complete with project, date and time, then helps you plan the day around what is already there.
HireKey
HireKey · Voice & Language · Productivity · LeaningHireKey is an AI job search copilot built by a former tech recruiter. It checks role fit, tailors resumes, runs interview practice and tracks every application.
Fluence
Yay Software (Yannick Huchard) · Writing · Voice & Language · ProductivityFluence is an Android app that reads PDFs, EPUBs, Word files and web articles aloud, follows each sentence as it speaks, and answers questions about the text on the phone itself.