AI Agent Testing: How to Build a Test Harness for Reliable Agents
Learn how to test AI agents that actually work: golden datasets, LLM-as-a-judge, trajectory evals, and the runtime governance that turns testing into reliability.
More than 40% of agentic AI projects will be scrapped by the end of 2027, according to Gartner, pointing to spiraling costs, unclear value, and weak risk controls. Read the postmortems and one pattern repeats: the team could never actually prove the agent was reliable. It demoed well, then quietly went wrong in production.
The difference between an agent that ships and one that gets killed is not a smarter model. It is a test harness: a repeatable system that measures whether your agent does the right thing, over and over, before and after it touches real data.
This guide shows you how to build one. You will get the six layers of a real AI agent testing harness, the metrics that matter, and the piece almost every tutorial skips: what to do when a test fails and your agent has already sent the email.
In a hurry? Spin up an agent you can watch work, free.
Why AI agents break traditional testing
Traditional software testing rests on one assumption: same input, same output. You write an assertion, it passes or fails, you move on. That assumption dies the moment you put a large language model in the loop.
AI agents are non-deterministic, multi-step, and tool-using. The same prompt can produce three different plans on three different runs. An agent might pick the right tool with the wrong arguments, or the wrong tool with a confident explanation of why it is right. It chains ten reasoning steps together, and a small error in step two quietly poisons step nine.
The state-space explosion
A function with two inputs has a handful of cases you can enumerate. An agent that reads an email, decides whether to refund a customer, calls a payments API, and writes a summary has a near-infinite space of possible paths. Twenty unit tests do not cover it. They barely scratch it.
Evals, not tests
This is why practitioners stopped saying "tests" and started saying evals. An eval is closer to an integration test for a probabilistic system. Instead of asserting one exact output, you score behavior across a set of cases and ask: is this good enough, often enough, on the dimensions I care about? The vocabulary shift is not cosmetic. It changes what you build.
Testing a deterministic function asks "is this output correct?" Evaluating an agent asks "is this behavior reliable, safe, and on-task, across the messy inputs it will actually see?"
The anatomy of an AI agent test harness
A test harness is not a single script. It is a layered system, each layer catching failures the others miss. Here are the six layers that matter, from cheapest to run to most revealing.
Layer 1: Golden datasets and success criteria
Before you write a single test, define what "good" looks like. Hand-craft 20 to 50 representative cases: real inputs paired with the ideal outcome or the acceptable range of outcomes. This is your golden dataset. It is tedious, and it is the highest-leverage hour you will spend, because every other layer scores against it. Anthropic makes the same point in its guidance on building effective agents: start simple, measure against real cases, and add complexity only when it earns its keep.
Layer 2: Unit and tool-call assertions
The deterministic parts of an agent can still be tested the old way. Did the agent select the correct tool? Did the arguments match the JSON schema? Is the output valid, parseable, within bounds? These assertions are cheap, fast, and belong in CI on every commit.
# A minimal tool-call assertion
result = agent.run("Refund the disputed charge on invoice #1042")
assert result.tool_called == "issue_refund"
assert result.args["invoice_id"] == "1042"
assert result.args["amount"] <= 500 # never refund above policy cap
assert result.requires_approval is True # sensitive action must pauseLayer 3: LLM-as-a-judge
Many agent outputs have no single correct string. A drafted reply, a summary, a plan: these need semantic scoring, not string matching. LLM-as-a-judge uses a separate model, guided by a rubric, to score each output. The research on LLM-as-a-judge found these judges can reach over 80% agreement with human preferences, on par with how often two humans agree, but it also documented real failure modes: bias toward longer answers, position bias, and inconsistency across runs. Use it, but calibrate it against human ratings on a sample before you trust it.
{ "role": "You are a strict evaluator of AI agent outputs.", "task": "Score the agent reply from 1 to 5.", "criteria": ["factual accuracy vs. the source data", "did it stay on task", "did it avoid taking an unapproved sensitive action", "tone matches a professional support reply"], "output": { "score": "1-5", "reasoning": "one sentence", "failed_criteria": ["list"] } }Layer 4: Trajectory evaluation
The final answer can be right for the wrong reasons. Trajectory evaluation grades the path, not just the destination: which tools the agent called, in what order, how many steps it took, whether it looped or backtracked. An agent that reaches a correct answer after nine wasted API calls is a cost and reliability problem hiding behind a green checkmark.
Layer 5: Simulation and scenario testing
Real users are messier than your golden dataset. Simulation runs your agent against synthetic users and multi-turn scenarios: the customer who changes their mind, the ambiguous request, the input designed to confuse. You are looking for where the agent drifts off-task across a conversation, not just on a single call.
Layer 6: Adversarial red-teaming
Finally, attack your own agent. Prompt injection hidden in a document, jailbreak attempts, inputs engineered to trigger a hallucination or an unsafe action. If your agent can spend money or send messages, a red-team pass is not optional. Better you find the hole than a stranger does.
What to measure: AI agent evaluation metrics
Layers tell you how to test. Metrics tell you what to record. A useful harness tracks at least these:
| Metric | What it tells you | Watch for |
|---|---|---|
| Task success rate | % of cases that hit the success criteria | The headline number, but never the only one |
| Tool-call accuracy | Right tool, right arguments | Silent drift after a prompt change |
| Trajectory efficiency | Steps and tokens per task | Correct answers that cost 5x too much |
| Safety violations | Unsafe or unapproved actions attempted | Any non-zero number is a release blocker |
| Latency and cost | Time and dollars per run | Spikes that only show under real load |
Offline versus online evals
Offline evals run against your golden dataset in CI, before you ship. Online evals run against real traffic in production, because no fixed dataset predicts everything the world sends you. The mature workflow is a loop: observe production, capture the failures, turn them into new offline cases, ship the fix, repeat. Which is impossible if you cannot see what your agent did in production, and that is the next problem.
AI agent observability: you cannot fix what you cannot see
Every layer above assumes you can inspect what the agent actually did: the reasoning, the tool calls, the arguments, the outcome. That is observability, and for agents it means tracing every step of every run, not just logging an error when something crashes.
Without traces, a production failure is a mystery. With them, it is a new test case. Good observability is what closes the loop between "something went wrong" and "here is the exact eval that now guards against it." If you want to go deeper on this, our guide to AI agent observability breaks down what to trace and why.
A beginner's guide to testing AI agents - PostHogSoftware testing used to be relatively straightforward. You wrote business logic, defined expected outputs, and verified them with unit, integration…
posthog.com
From testing to reliability: why the harness is not enough
Here is the uncomfortable truth every eval-focused article skips. A test harness tells you when your agent misbehaves. It does nothing to contain the misbehavior that slips through.
And something always slips through. Your golden dataset is finite. Production is not. The gap matters far more for agents than for chatbots, because an agent does not just talk. It acts. It spends money, sends emails, updates records, deletes rows. A failure you did not catch in testing becomes a refund sent to the wrong customer, an email to your whole list, a database wiped.
Testing catches failures before you ship. Governance catches the ones that slip through when your agent can actually cause damage. Reliable agents need both.
Runtime governance turns testing into reliability
Three controls close the gap between "tested" and "trustworthy":
- Human-in-the-loop approvals. For any sensitive action, the agent pauses and asks a person before it acts. Your eval said "should require approval," runtime governance enforces it. Our deep dive on human-in-the-loop AI agents covers the patterns.
- Least-privilege permissions. Scope every tool and credential so tightly that even a failed eval cannot become a production incident. An agent that cannot exceed a refund cap cannot refund $50,000.
- Observability and audit trails. Every action traced and replayable, so an online eval doubles as a governance record you can review after the fact.
Human-in-the-Loop AI Agents: The Complete Guide to Building Agents You Can Actually Deploy
Human-in-the-loop AI agents pause on high-stakes actions to get human approval. Here are the approval gates, confidence thresholds, and escalation patterns that make agents production-ready.
Why flowchart tools, chatbots, and DIY frameworks fall short
If testing agents is this involved, why not reach for something simpler? Because the simple options solve a different problem.
Zapier, Make, and n8n
These are deterministic flowcharts. They are excellent at "when X happens, do Y," and they are not agents. There is no reasoning to test, which sounds like a feature until the input falls outside the branches you wired, and the flow snaps off-script with no judgment to fall back on. You are not testing an agent. You are maintaining a brittle diagram.
Chatbots
A chatbot is conversational, not operational. It answers, it routes tickets, it does not take real actions on your systems. There is nothing to red-team at the action level because there are no actions. The moment you need the thing to actually do the work, a chatbot is the wrong tool, and its "testing" story does not transfer.
DIY agent frameworks
Stitch together a framework, an eval library, a tracing tool, and a homegrown approval queue and you can absolutely build a real harness. You will also own all of it forever: the glue code, the governance layer, the on-call when the approval webhook breaks at 2 a.m. For a lot of teams, that maintenance burden is exactly the cost Gartner is counting when those projects get canceled.
Rerun: test, deploy, and govern reliable agents
A test harness proves your agent works. Rerun makes sure it keeps working when it acts, with the governance layer built in instead of bolted on.
Rerun is the platform that lets you run AI agents you can actually watch work. You build an agent in minutes, connect your tools, then watch every action live on a dashboard anyone can read. That transparency is not a nice-to-have for testing. It is the observability layer your harness needs, without you building it.
What that gives you when reliability is the goal:
- Agents that act, with approval gates. Before any sensitive action, the agent stops and asks. Approve from the app or Slack, and it resumes exactly where it paused.
- Least-privilege by default. Scope every tool and credential so a bad run cannot become a production incident.
- Full observability and audit trails. Every run, token, tool call, and handoff is traced and replayable, so your online evals and your governance record are the same thing.
- No code, no black box. If you can describe the job, you can build the agent, and you can see everything it does.
How Rerun compares
| Capability | Zapier / Make / n8n | Chatbot builders | DIY frameworks | Rerun |
|---|---|---|---|---|
| Handles non-deterministic reasoning agents | No | Partial | Yes | Yes |
| Agents that take real actions | Partial | No | Yes | Yes |
| Built-in observability and tracing | Partial | Partial | Partial | Yes |
| Human-in-the-loop approvals | No | No | Partial | Yes |
| Least-privilege permissions | No | No | Partial | Yes |
| No flowcharts to wire or maintain | No | Yes | Partial | Yes |
| Maintenance burden | Low but rigid | Low | High | Low |
Your agent-testing checklist
Before you call an agent production-ready, walk this list:
If you want the deployment side of this, our guide on how to deploy AI agents picks up where testing ends, and AI agent guardrails goes deep on the safety layer.
The bottom line
Models are getting better on their own. Reliability is not something you get for free with the next release. It is engineered, through a test harness that measures behavior and a governance layer that contains the failures your tests miss.
That is the line between the agent projects that ship and the 40% that get canceled. Test hard, then deploy on infrastructure where approvals, least privilege, and observability are the default, not a side project.
Frequently asked questions
What is AI agent testing?
AI agent testing is the practice of evaluating autonomous, non-deterministic AI systems to verify they behave reliably, safely, and correctly across real-world conditions. Because agents reason over multiple steps and call external tools, testing goes beyond pass/fail unit tests and uses layered evals: golden datasets, tool-call assertions, LLM-as-a-judge scoring, trajectory evaluation, simulation, and adversarial red-teaming.
How do you test an AI agent before production?
Build a test harness in layers. Start with a golden dataset of 20 to 50 real cases and clear success criteria. Add unit and tool-call assertions in CI, then LLM-as-a-judge for open-ended outputs, trajectory checks on the path the agent takes, simulation against synthetic users, and a red-team pass for prompt injection and unsafe actions. Then keep evaluating in production with online evals and full tracing.
What is the difference between AI agent testing and evaluation (evals)?
They describe the same goal from two angles. Traditional testing assumes deterministic outputs and asserts an exact result. Evals are the version adapted to probabilistic agents: instead of one correct string, you score behavior across many cases on dimensions like task success, tool accuracy, cost, and safety. In practice, agent testing is done through evals.
What is LLM-as-a-judge and is it reliable?
LLM-as-a-judge uses a separate language model, guided by a rubric, to score agent outputs that have no single correct answer, such as a drafted reply or a summary. It is useful and scalable, but it has failure modes: bias toward longer answers, position bias, and run-to-run inconsistency. Calibrate it against human ratings on a sample before trusting it, and pair it with deterministic assertions.
What metrics should you use to evaluate an AI agent?
Track task success rate, tool-call accuracy, trajectory efficiency (steps and tokens per task), safety violations, and latency and cost. Task success is the headline, but never the only number: an agent that reaches the right answer through wasted steps or an unapproved action is still a reliability problem. Run these metrics offline against a golden dataset and online against real traffic.
Can you fully test a non-deterministic AI agent?
No. A finite test suite cannot cover the near-infinite space of inputs and reasoning paths, so some failures always reach production. That is why testing is necessary but not sufficient. Reliable agents pair a test harness with runtime governance: human-in-the-loop approvals on sensitive actions, least-privilege permissions, and full observability, so the failures your tests miss are contained instead of catastrophic.
How is agentic testing different from testing AI agents?
The terms are often confused. Testing AI agents means evaluating agents for reliability and safety, which is the subject of this guide. Agentic testing usually refers to the opposite: using AI agents to perform software QA, such as generating or running test cases on a traditional application. This article is about the former, making your own agents reliable enough to deploy.
Written by
Clément Janssens
