Relari: Testing and Simulation Stack for GenAI Systems
Relari, Inc. · Coding · Chatbot
Relari is a testing and simulation platform for GenAI systems, pairing synthetic test data with modular LLM evaluation metrics so teams can catch pipeline failures before users do.
About Relari: Testing and Simulation Stack for GenAI Systems
What Is Relari
Relari is a testing and simulation stack for GenAI applications, built by Relari, Inc., a startup that went through Y Combinator's Winter 2024 batch. The pitch is simple. Self-driving teams don't wait for a car to crash on real roads before they find a bug, they drive simulated miles instead. Relari applies that idea to LLM apps: generate the traffic, run the pipeline against it, and score what breaks.
The product covers three jobs that usually live in separate tools. It builds synthetic test sets that imitate real user behavior, it scores every module in a pipeline with open-source metrics, and it watches production traffic for quality drops. The open-source half of that stack, continuous-eval, is a Python package under Apache 2.0 with more than 30 metrics for retrieval, generation, code, classification and agent tool use. Anyone can install it with pip and run it without paying Relari a cent.
The limits matter as much as the features. Relari assumes you already have a working pipeline and some definition of correctness, because a contract written in plain English still needs a human to write it. Teams without labeled examples or production feedback will get less out of the custom evaluators. The hosted platform is also no longer actively promoted: the company has since shifted its public focus toward agent building tools, and relari.ai no longer serves the evaluation product. The open-source package is still on GitHub and PyPI, with v0.3.14 as the last release in January 2025.
Getting Started
- Create a free account on the Relari platform, or skip the account and install continuous-eval with pip if you only need offline scoring.
- Describe your pipeline as connected modules, mapping each step's input and output so the runner knows where to attach metrics.
- Pick metrics for each module, such as retrieval precision and recall for the retriever and answer correctness for the final LLM call.
- Generate a synthetic test set from your docs or sample questions, then run the evaluation and read the per-module scores.
- Add pass or fail thresholds as tests, and wire the run into your release checklist so a regression blocks the deploy.
Product Information
A quick look at Relari: Testing and Simulation Stack for GenAI Systems's pricing, supported platforms, and performance.
Best for
The users, tasks, and scenarios where this tool fits best.
Users
- AI engineers building LLM apps
- RAG developers
- QA and platform teams at larger companies
- Researchers and students
Tasks
- Building a golden dataset without hand labeling
- Finding the failing step in a broken answer
- Stress testing an AI agent before launch
- Setting up a quality gate in CI
Scenarios
- A support bot starts answering refund questions wrong after a prompt edit
- A compliance copilot must prove it cites the right policy
- An enterprise search team wants fewer hallucinations
Key features
Module Level Evaluation for Pipelines
Relari treats an application as a graph of modules, not one big black box. You declare the retriever, reranker and generator separately, attach different metrics to each, and the runner returns scores per node. When an answer is wrong, the scoreboard points at the step that broke instead of blaming the whole system. That's the difference between a number you can act on and a number you argue about.
More Than 30 Open-Source Metrics
The continuous-eval library ships metrics for retrieval, text generation, code generation, classification and agent tool use. Deterministic checks, semantic similarity and LLM based judging sit side by side, so you can mix a cheap exact match with a model graded rubric in the same run. Custom metrics take a criteria and a rubric plus typed arguments, which is enough to build a judge for almost any policy you can write down.
Synthetic Test Data Generation
Public benchmarks are polluted and rarely match your domain, and manual annotation is slow. Relari generates test sets from your documents and sample interactions, including adversarial and corner case prompts. The sales pitch is coverage: thousands of realistic queries in an afternoon instead of a curated set of forty.
Human Aligned Evaluators
Generic judges disagree with your users. Relari trains custom evaluators on your own feedback data and reports alignment above 90 percent with human raters in its research. Those evaluators then grade new outputs, which closes the loop between what production users complain about and what your offline suite measures.
Agent Contracts and Behavior Specs
Later versions of the platform moved toward contracts: short natural language statements of what an agent must and must not do. A contract turns vague intent into checkable behavior, so AI agent testing covers refusals, tone and scope, not just whether the final string looks plausible. It's the most interesting idea in the stack, and the least documented one.
Production Monitoring and Feedback Loop
Offline scores drift away from reality once real traffic arrives. Relari's online monitoring samples live conversations, scores them with the same metrics and feeds the failures back into the test set. Teams that keep that loop running get a test suite that ages with the product instead of going stale.
Pros and cons
Pros
- The open-source core is genuinely free and permissively licensed, so you can evaluate a pipeline today and decide about the paid tier later.
- Module level scoring answers the question most teams actually ask, which is where the failure came from, not whether the answer was good.
- Metric breadth covers RAG evaluation, code and agent tool use in one library, which saves wiring three separate tools together.
- Synthetic data generation removes the slowest part of setting up an LLM evaluation workflow, which is finding a few hundred realistic questions.
Cons
- The hosted platform is hard to buy right now. The company has moved its public attention to agent tooling, and the pricing page is gone, so you'll work from the open-source package and the docs.
- Custom evaluators need your own feedback data. Without labeled production examples, the human alignment claims stay theoretical for your use case.
- Setup assumes real engineering effort. Describing a pipeline as modules and writing meaningful thresholds is a day of work, not a click, and small prototypes rarely justify it.
- Documentation lags the product. Some newer ideas, contracts in particular, are easier to understand from blog posts and the YC launch thread than from the reference docs.
Frequently asked questions
Relari simulates, tests and scores GenAI applications. It generates synthetic inputs that imitate users, runs your pipeline against them and reports scores for each module, so you can tell whether a bad answer came from retrieval, ranking or generation.
Related content
Explore related tools, skills, and articles for Relari: Testing and Simulation Stack for GenAI Systems.
Relari: Testing and Simulation Stack for GenAI Systems Alternatives
Base Code
Base44 (Wix) · Coding · ProductivityBase Code is a shared cloud development environment from Base44. Teammates describe changes to an existing GitHub codebase in chat, preview them live and send them back as pull requests.
Google EmbeddingGemma 2
Google DeepMind · Writing · Image · CodingGoogle EmbeddingGemma 2 is a 740M-parameter open model that turns text, code, images, audio, and video frames into vectors in one 768-dimension space, running on phones and laptops.
GitGlow
GitGlow · Coding · ProductivityGitGlow is a local-first desktop Git client that lets independent developers review their own code and their AI agents' changes in one place before shipping.