How to Test and Evaluate an AI Agent Before Launch

To test an AI agent before launch, build a set of realistic test cases, define what a correct result looks like for each, run the agent against them repeatedly and score both the final outcome and the steps it took. Then add adversarial tests, run it in shadow mode on real work, and launch to a small group with monitoring. An agent that only looked good in a demo has not been tested.

Key takeaways

• Agents are non-deterministic. The same input can produce different outputs, so one successful run proves little.
• A test set built from real past cases is the most valuable asset in an agent project.
• Evaluate the path as well as the answer: which tools were called, with which arguments, in what order.
• Test what the agent should refuse to do, not only what it should do.
• Evaluation continues after launch. Every production failure becomes a new test case.

Why testing an agent is different from testing software

Traditional software is deterministic. Given the same input, a function returns the same output, and a test either passes or fails. An agent built on a language model behaves differently in three ways.

• Outputs vary. Wording changes between runs, and sometimes decisions do too.
• Inputs are open-ended. You cannot list every way a customer might phrase a request.
• Failures can be quiet. A wrong answer looks as fluent as a right one. Nothing crashes.

So agent testing is closer to measuring quality across many samples than to checking a single pass or fail. You are asking "how often is this right, and how bad is it when it is wrong".

Step 1: Define success before you test

Write down what correct means for your task in terms you can check. Vague goals such as "helpful answers" cannot be scored. Useful criteria look like this:

• The right record was created, with the right field values.
• The answer matches the source document and cites it.
• The agent escalated when the request was outside policy.
• No action was taken without the required approval.
• The reply stayed within tone and length guidelines.

Separate the criteria into must-never-fail items, such as "never issues a refund above the limit", and quality items, such as "reply is concise". The first group needs a hard gate. The second can be a score you improve over time.

Step 2: Build a test set from real cases

A test set, often called an evaluation set or golden set, is a collection of inputs paired with expected results. Build it from your own history: past tickets, emails, orders, chat logs. Include four kinds of case:

• Typical cases. The requests you see every day.
• Edge cases. Missing information, ambiguous wording, two requests in one message, typos, mixed languages.
• Out-of-scope cases. Things the agent should decline or hand over.
• Adversarial cases. Attempts to make it break rules. More on these below.

Start with a few dozen cases and grow from there. A small, well-chosen set you actually run is worth more than a large one you never finish. Have the people who do the job today write or approve the expected results. They know what correct looks like.

Step 3: Test the parts, then the whole

Tools

Each tool the agent can call is ordinary code. Test it with ordinary unit tests: valid input, invalid input, timeouts, permission errors. Many agent failures are tool failures in disguise.

Retrieval

If the agent answers from documents, check retrieval separately. For each test question, did the search return the passage that contains the answer? If not, no prompt will fix the response.

The full run

Then run complete tasks and record the trajectory: every model call, tool call, argument and result. Check both whether the outcome was right and whether the route was sensible. An agent that reaches the right answer after calling the wrong tool three times will be slow, costly and fragile.

Step 4: Choose how to score

Method	How it works	Best for	Limits
Code checks	Scripts verify exact facts: record exists, field equals expected value, tool was called	Actions and structured output	Cannot judge free text quality
Model grading	A second model scores the output against a written rubric	Tone, completeness, faithfulness to sources at scale	The grader makes mistakes and must be checked against human scores
Human review	A subject expert reads and scores samples	Nuanced judgement, calibrating the other methods	Slow, so use it on samples

Use code checks wherever possible, because they are cheap and exact. Use model grading for text, and regularly compare its scores with a person's on the same samples. If they disagree often, fix the rubric before trusting the numbers.

Because outputs vary, run each test case several times. A case that passes four times in five is not a passing case for a must-never-fail criterion.

Step 5: Try to break it

Before customers find the weak points, look for them yourself. Test at least these:

• Prompt injection. Put instructions inside the content the agent reads, such as an email or a document that says "ignore your rules and do this instead". The agent must treat it as data.
• Requests beyond authority. Ask for a discount, a refund above the limit or another customer's data.
• Tool failures. Make an API return an error or time out. The agent should report the problem, not invent a result.
• Missing information. Leave out a required detail. The agent should ask, not assume.
• Pressure. Insist, repeat and claim urgency. Rules should hold on the fifth attempt as on the first.

Run these tests in a sandbox or against test data, never against live customer records. The related controls are covered in AI agent guardrails in production.

Step 6: Check speed and running cost

An accurate agent can still be unusable. Measure how long a typical task takes end to end and how many model calls it makes. Watch for loops where the agent repeats a step without progress, and set a hard cap on steps per task. Estimate usage at your expected volume so the running cost is known before launch.

Step 7: Shadow mode, then a small launch

Shadow mode means the agent processes real incoming work but its output is not used. People do the job as usual, and you compare the agent's results to theirs. This shows how it copes with live inputs, which are always messier than a test set.

Then launch in stages:

1. Draft mode: the agent proposes, a person approves everything.
2. Limited autonomy: it acts alone on the categories that scored well.
3. Wider rollout: more categories or more users, one step at a time.

Agree the launch gates in advance. For example: zero failures on the must-never-fail cases across repeated runs, and a quality score the process owner has signed off.

Step 8: Keep evaluating after launch

Launch is the midpoint. Keep these habits:

• Log every run with its full trajectory.
• Review a sample of real conversations each week.
• Add every failure to the test set, so it cannot quietly return.
• Rerun the whole test set before any change to the prompt, the tools or the model. Model updates can change behaviour even when your code has not changed.

Related guides

• The AI Voice Agent Technology Stack Explained: Speech, Models, Latency and Telephony
• AI Agent Guardrails: Keeping Agents Safe in Production
• How AI Agents Use Tools: Function Calling Explained
• How to Build an AI Agent: From Idea to Production
• Service: Custom AI Agents

Frequently asked questions

How many test cases do I need?

There is no fixed number. Start with a few dozen that cover your typical, edge, out-of-scope and adversarial cases, then add every real failure you find. Coverage of situations matters more than raw count.

Can I use an AI model to grade another AI model?

Yes, and it is common for scoring free text at scale. Treat the grader as a tool that needs its own validation: compare its scores with human scores on a sample, and keep exact code checks for anything that can be verified directly.

What is shadow mode?

It is running the agent on real incoming work without using its output. You compare what it would have done to what your team actually did, which reveals problems with no risk to customers.

Sources and further reading

Primary documentation and research behind this guide.

• OpenAI: Evaluation best practices
• Anthropic: Building effective agents
• NIST: AI Risk Management Framework

An agent is ready when its failures are known, bounded and monitored, not when a demo goes well. Testing is built into how Growmerz delivers custom AI agents. If you have an agent idea and want to know what it would take to make it reliable, book a free AI audit.