Langfuse

Langfuse captures hierarchical traces of every LLM call, tool use, and retrieval step, then lets you filter by user, session, cost, latency, or custom metadata. When an agent misbehaves, this is where you find the exact step that broke.

You can score outputs with an LLM judge, simple heuristic functions, or human review. Evaluators run on live production data or inside experiments. Quality checks happen continuously, not once before launch.

Prompts live outside your code with versioning, one-click deployments, and rollbacks. That keeps prompt edits from turning into a deploy, which is a real time saver on fast-moving teams.

The playground lets you test prompts against real production inputs and compare models side by side. It's useful for checking whether a cheaper model can handle a task before you commit to switching.

Define test cases, run experiments, and compare results to see which configuration wins. This turns "I think it's better" into a measurable before-and-after.

Dashboards track token spend, latency, and quality together, with automated alerts when something drifts. Handy. Teams often use this to catch a runaway model cost before the bill arrives.

A public API, a metrics API, and exports to S3 or a data warehouse let you build on top of Langfuse. Think billing from tracked LLM costs, or LLM analytics correlated with user behavior. It's a solid fit if you already run a warehouse.