LangWatch

Every LLM call, tool call, retrieval, and evaluation lands as a span inside one trace, with nested spans and an execution graph for agent runs. Token and cost accounting is automatic per provider, prompt, and model. Traces group into sessions by thread id, so you can read a whole chat instead of a single request. Text, image, and audio payloads are captured, which matters when an agent handles screenshots or voice notes.

AI agent testing here starts with a written scenario, and an LLM plays the user by generating realistic messages from it. The run covers multi-turn dialogue instead of a single prompt. A judge agent scores each turn and can call pass or fail at any point. Tool-call verification checks that the right tool fired at the right moment across a long conversation. There's a visualizer for replaying a run step by step and pausing mid-conversation to annotate it. Voice testing supports ElevenLabs, OpenAI Realtime, Twilio, Pipecat, and Gemini Live, with time-to-first-byte, p50 and p95 latency, plus background noise and interruption injection. Adversarial runs use Crescendo escalation and refusal detection to find weak spots before a user does.

Built-in evaluators cover RAGAS metrics, hallucination, toxicity, PII detection, and LLM-as-a-judge scoring. You can run offline experiments from the SDK or from a no-code wizard in the UI, then compare runs on quality, cost, and latency. Production monitors score live traffic and alert when quality drops. Scoring works on a whole thread as well as a single message, which fits support bots and multi-step agents. Datasets can be built from real traces, generated with AI, or uploaded with image inputs, and an annotation inbox handles human review. Custom evaluators can be defined once and shared across projects, and the same checks can run inline as guardrails.

One endpoint speaks both the OpenAI and Anthropic APIs, and routes to OpenAI, Anthropic, Google, Bedrock, Mistral, or any other provider with your own keys. Each caller gets a virtual key with its own budget, set per person, team, or customer. The company claims 11 microseconds of overhead. Every request that passes through lands as a trace in the same project, so gateway traffic and SDK traffic sit in one place.

Prompts live as structured objects with messages, inputs, and config, editable from code, the UI, or the API. Liquid templating adds variables and logic. GitHub sync lets a team review prompt changes like any other pull request, and tags mark staging and production versions. A playground runs variants side by side across models, and A/B tests report quality, cost, and latency for each version. Webhooks and Slack notifications fire when a prompt changes.

A single command, npx langwatch claude, brings Claude Code, Codex, Copilot, Gemini CLI, and opencode onto the same observability standard. Sessions show tokens, cost, and cache usage per engineer and per repository, which answers the question of where AI spend actually goes. The AI governance side adds an inventory of every AI tool and provider in use, anomaly rules on activity, and an audit log for security reviews.