In 2021, GPT-3 solved roughly 35% of the grade-school math problems in the GSM8K benchmark. Today, GPT-5.3 Codex scores 99% on the same test. The model got better, sure. But the benchmark stopped doing its job long before that: once every leading model clears 90%, a test that used to separate good from great tells you nothing at all. LangChain tracks the whole arc, and it is a warning about a much bigger mistake. Teams that judge their AI by benchmarks alone end up shipping systems that look great on paper and break on real users.
LLM evaluation is the practice of measuring whether a model, or an application built on one, actually works. It has two layers, and confusing them is the single most common failure in AI engineering today. A benchmark scores a model in isolation. An eval scores your system on your data, your failure modes, and your users.
This guide covers both layers. You will learn the difference between benchmarks and evals, the five evaluation methods and when to use each, how LLM-as-a-judge works and where it goes wrong, how to evaluate RAG systems and agents rather than just single answers, and how to turn evaluation from a pre-launch checkbox into a continuous production loop.
In a hurry? Start your first coworker free for 5 days and see the observability and approval layer evaluation needs.
What Is LLM Evaluation? (And Why It Matters)
Model evaluation vs. application evaluation
Start with the distinction every serious source makes. IBM draws the line as clearly as anyone: model evaluation asks "does the model work," while system evaluation asks "does the application built around it work." They are different questions with different answers.
Model evaluation measures raw capability: can the model understand language, follow instructions, reason, and generate quality text across tasks. It runs on shared benchmarks and leaderboards, and it answers one question: which model should I start from.
Application evaluation measures your system: the prompt, the retrieval, the tools, the guardrails, the latency, the cost, and the end-to-end task. A strong model can still produce a weak application when the surrounding system supplies bad context, picks the wrong tool, or never completes the action the user asked for.
A benchmark tells you whether a model looks good on paper. An eval tells you whether your system works in production. Both matter. Only one decides whether to ship.
Why LLM evaluation matters
Most teams understand that software needs tests. What trips them up is that LLMs break the two assumptions those tests rely on.
First, outputs are not deterministic. The same input can produce several "correct" answers, so a string-match test that would pass on traditional code fails on a good answer that happens to use different words.
Second, a response can be fluent and completely wrong at the same time. A support agent can answer with total confidence that an address was updated when it actually called the wrong tool, ignored an error, or never called anything at all.
Evaluation exists to catch those failures before users do, and to leave evidence of what was tested when a regulator, an auditor, or a nervous executive asks. It is how a team proves its system works rather than insisting it does.
The Two Layers: Benchmarks vs. Evals
Popular LLM benchmarks and what they measure
Benchmarks are standardized test suites that score a model's raw ability on fixed tasks. Each probes a different slice of capability:
| Benchmark | What it measures | Why it matters |
|---|---|---|
| MMLU | Multiple-choice knowledge across 57 subjects | The default "is this model generally smart" check |
| HumanEval / SWE-bench | Code generation and functional correctness | The standard for coding models |
| GSM8K | Grade-school math reasoning | The classic reasoning test, now saturated |
| TruthfulQA | Resistance to confident misinformation | Hallucination and honesty |
| MT-Bench | Multi-turn instruction following | Judge-verified conversational quality |
| Chatbot Arena | Head-to-head human preference votes | What real users actually prefer |
| τ-bench / WebArena / GAIA | Tool use and agent task completion | How well a model acts, not just answers |
You can watch these numbers move in real time on the Hugging Face Open LLM Leaderboard and the Stanford HELM suite.
Why benchmarks don't predict production
Three failure modes, in order of how quietly they bite.
Saturation. Once the leading models all land within a few points of a perfect score, the benchmark has hit its ceiling. By 2024, GPT-4o, Claude 3.5, and Gemini 1.5 all scored above 90% on GSM8K. A test where everyone gets an A no longer separates the top of the class.
Contamination. Test questions leak into training data, and scores start to reflect memorization instead of reasoning. The clearest evidence is GSM1k, a fresh benchmark built to mirror GSM8K. Zhang and colleagues found several model families scored worse on the uncontaminated set, a sign their GSM8K numbers were partly memorized. The worst model dropped 13 points.
Isolation. A benchmark cannot see your prompts, your retrieval corpus, your tools, your policies, or your users. A legal assistant succeeds by returning verifiable citations; a support bot succeeds by resolving the complaint. A borrowed test suite, however rigorous, answers neither.
Evals: tests on your own data
An eval is a test you write against your inputs and your failure modes. It runs on a golden dataset of real or representative traces, labeled pass or fail, and it scores the behavior you actually care about. Where a benchmark asks "is this model capable in general," an eval asks "does this system do the thing I shipped it to do." LLM evaluation, done right, is mostly the second layer.
LangChain's guide puts the split in one line: a benchmark is a borrowed test suite, written and validated for someone else's program. Your application has nothing to borrow, so you have to build your own.
Core LLM Evaluation Metrics & Methods
Five methods, each trading coverage against trust. You need more than one, because no single method catches every failure mode.
Reference-based metrics
Compare outputs against a known-good answer. BLEU and ROUGE measure word overlap, BERTScore uses embeddings to accept paraphrase, and F1 or exact match grade extraction tasks. These work for structured jobs with a gold answer: classification, extraction, translation. The tradeoff is that they measure surface similarity, and an answer can be correct while sharing no words with the reference.
Reference-free / intrinsic metrics
Score properties of the output itself, like perplexity or fluency. Useful as a cheap first screen when you have no ground truth. The tradeoff is that perplexity tells you how well the model predicts text, not whether the text is correct for your task.
Deterministic code-based checks
Rules you express in code: JSON validity, schema compliance, regex, tool-argument validation, latency and token limits. These are the cheapest, fastest, and most reliable signal, and they never disagree with themselves. The tradeoff is that they can verify a field exists, but not whether an open-ended answer is genuinely helpful.
LLM-as-a-judge
Use a model with an evaluation prompt and a rubric to grade outputs. This scales quality review past what humans or code can cover, and it is the workhorse for semantic qualities like helpfulness, tone, and faithfulness. The tradeoff is that a judge is itself a failure mode until you calibrate it. More on this below.
Human annotation
Expert reviewers supply the ground-truth labels. The slowest and most expensive signal, and the most reliable. Its real job is to calibrate the other four: the golden data a judge is checked against comes from humans, not from another model.
| Method | What it checks | Best for | Key tradeoff |
|---|---|---|---|
| Reference-based | Overlap with a gold answer | Structured tasks | Misses meaning |
| Reference-free | Intrinsic properties | Screening | Says little about success |
| Code-based | Deterministic rules | Formats, limits, tool args | Cannot judge nuance |
| LLM-as-a-judge | Rubric-graded quality | Open-ended quality at scale | Bias until calibrated |
| Human annotation | Expert judgment | Golden labels, high-risk | Slow and expensive |
LLM evaluation tools and frameworks
You do not have to build an LLM evaluation harness from scratch. The ecosystem has split into a few clear options:
| Tool | What it is | Best for |
|---|---|---|
| LangSmith / LangChain | Agent platform with evals and tracing built in | Teams on LangGraph or LangChain |
| Arize Phoenix / AX | Observability plus evals, open source and managed | Tracing-heavy agent teams |
| Evidently | Open-source LLM eval library with 25M+ downloads | Self-hosted judges and checks |
| Braintrust / Langfuse | Eval and experiment platforms | CI-integrated model and prompt testing |
The tool is the least important part of the decision. The method mix, the dataset, and the loop you run around them matter far more.
LLM-as-a-Judge, Explained
How it works
LLM-as-a-judge works because assessing text is easier than generating it. The generator has to hold the whole task, the context, and the user in mind at once. The judge has one job: read a response against a rubric and return a label or a score. As Evidently AI puts it, it is easier to critique than to create.
There are two shapes. Direct scoring gives a single output a grade against criteria. Pairwise comparison shows two responses and asks which is better, which is how you compare models or prompts during development.
A judge is only as good as the evidence you hand it. A hallucination judge for a RAG answer needs the retrieved context alongside the response, not just the answer. Judge the answer alone and the judge is guessing.
Here is a real evaluation prompt for a faithfulness judge, the kind you would run against a RAG system:
{
"task": "Evaluate whether the response is grounded in the context",
"inputs": ["question", "retrieved_context", "response"],
"rubric": {
"faithful": "Every claim in the response is supported by the context, with no invented details",
"not_faithful": "The response adds information not in the context, or contradicts it"
},
"output": "one label: 'faithful' or 'not_faithful', plus a one-sentence reason"
}Techniques that make judges accurate
A handful of prompting practices move the needle measurably. Use binary or low-precision labels instead of a 1 to 10 scale, because "correct vs. incorrect" is far more consistent than "is this a 7 or an 8." Define each label in the prompt, the way you would brief an intern. Split complex criteria into separate judges rather than asking one judge to grade accuracy, tone, and safety at once. Add a few labeled examples when the criteria are subtle. Ask the judge to reason step by step before the verdict, and set a low temperature so it gives consistent answers.
One practice matters more than any of these: calibrate the judge against human labels before trusting it. The paper that gave the technique its name, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng et al., found strong judges reached over 80% agreement with human evaluators, the same level of agreement humans reach with each other. That is the bar, and you only reach it by testing the judge on labeled data and fixing where it disagrees.
Known biases and how to handle them
Even top judge models are not neutral. A December 2025 study of judge reliability, Are We on the Right Way to Assessing LLM-as-a-Judge by Feng et al., found that the strongest judges, Gemini-2.5-Pro and GPT-5, failed to hold consistent preferences on nearly a quarter of difficult cases. Three biases recur:
- Position bias. The judge favors whichever response is listed first. Fix: swap the order and average.
- Verbosity bias. The judge prefers longer answers even when they are not more accurate. Fix: score length separately, or prefer direct scoring over pairwise.
- Self-enhancement bias. The judge favors text generated by the same model family. Fix: use a judge from a different provider than the generator.
A judge is not ground truth. It is a fast approximation of it, and it stays useful only while you keep checking it.
Evaluating Beyond the Single Answer: RAG and Agents
RAG evaluation
Retrieval-augmented generation fails in two places, and they need separate evals. Evaluate retrieval on its own: did the system find documents that were relevant, sufficient, and well ranked? Then evaluate generation on its own: was the answer faithful to the context, did it answer the question, and do the citations actually support the claims?
Splitting them is what makes failures actionable. If retrieval scores low, you fix the embeddings, chunking, or reranker. If the context is right but the answer is unsupported, you fix the prompt or the model. If both score well but users stay unhappy, you have the wrong definition of the task. Our agentic RAG guide covers the retrieval side in depth.
Agent & trajectory evaluation
Agents break the "score the final answer" assumption entirely. An agent can hallucinate a tool call and still get lucky on the result. It can pull irrelevant documents and still stitch together a passable answer. It can return the right reply for the wrong reason, and a final-answer judge will never see the difference.
So you evaluate the trajectory, not just the output. Did the agent select the right tool? Extract the right parameters without inventing any? Use the tool result correctly? Recover when a call failed? Reach the expected final state rather than only describing it? The sequence of decisions is the evidence, and in regulated work it is also the audit trail proving the agent followed protocol.
Our guide to AI agent testing goes deeper on building these suites, and AI agent observability covers the traces you need to inspect them.
Multimodal evaluation (brief)
The same principles scale to systems that read images, documents, charts, and tables. You still check grounding, completeness, and correctness. You just add source evidence: is the answer supported by the image, page, or table it references, and does the text agree with the non-text source? Deterministic checks handle schemas and bounding boxes; a vision-capable judge handles whether a summary matches a chart.
From Pre-Launch to Production: Evaluation as a Continuous Loop
Most teams treat evaluation as a gate: run the evals, get a green check, ship, move on. That is the moment the whole system starts to rot, because production changes faster than any dataset.
Offline vs. online evals
Offline evals run before release on a curated dataset: known cases, edge cases, and the failures collected from the last version. They answer "did this change break anything we know about."
Online evals run after release on real traffic. They answer "what is breaking that we did not know to test for." Some run as guardrails, blocking a bad output in real time. Most run asynchronously on a sample, flagging patterns for review without adding latency.
The insight that ties them together: the same evaluator often runs in both places. The faithfulness judge that scores a curated dataset before release scores a sample of production traces after it. Same rubric, different data source.
Turn production failures into regression tests
The loop is what makes evaluation pay off. Observe, categorize, add the failure to the dataset, fix, verify, re-release, observe again.
Evaluating LLMs and Agents: Benchmarks, Evals & GuardrailsLearn how to evaluate LLMs and AI agents with practical benchmarking methods, LLM-as-judge, deterministic checks, and human feedback workflows.LangChain's own Chat LangChain assistant is the cleanest named example. When the company shipped Deep Agents, users came asking how to build with it, but a stale guardrail kept blocking the question. Monitoring clustered the blocked requests under one failure mode, the traces showed the guardrail firing on legitimate questions, and the fix was a prompt update. Without the loop, that signal arrives days later as scattered complaints.
Every failure caught this way belongs in a dataset. A dataset does not need to be large to be effective. It needs to represent the paths and failure modes that actually matter.
Governance, human-in-the-loop & observability
Evaluation is not just an engineering habit. It is the evidence layer for governance. In high-stakes or regulated work, the scored trajectory is what you show to prove an agent followed the rules: what it was asked, what it retrieved, which tools it called, what it decided, and how a human reviewed the uncertain cases. Arize's guide walks through the governance angle, including how evaluation supports EU AI Act obligations for high-risk systems and SR 11-7 model-risk guidance in financial services.
That is also where the evaluation loop needs an operations layer to live on. An eval in a notebook is a snapshot that goes stale the day traffic changes. Evaluation only becomes continuous where the agent actually runs: a place that watches real traffic, routes uncertain outputs to a human reviewer, and feeds corrections back as the next test case.
This is the layer Rerun provides. Rerun is not an evaluation framework and not a benchmark suite. It is the operations layer where agents run under governance, with observability, approvals, and a human in the loop, so the evaluation loop keeps turning in production instead of stopping at the release gate. On Rerun, these agents are called coworkers.
How to Build an LLM Evaluation Strategy (start small)
You do not need every metric before you ship. You need the most important behaviors measured well enough to guide decisions.
Define the quality bar
Write down what the application must do and what it must never do. Separate "be helpful" from the concrete rules: resolves the request, follows the refund policy, never invents an order number, escalates when uncertain.
Build a golden dataset
Start with 50 real traces, or 50 representative inputs if you have not shipped yet, and label each pass or fail yourself. Small and representative beats large and duplicated. That labeled set seeds your first offline suite and calibrates your first judge.
Match evaluators to failure modes
Start from the failure you fear most, not from the tool. Hallucination gets a faithfulness judge. Malformed JSON gets a code check. Tone and on-brand phrasing get a rubric judge. High-stakes correctness gets a human reviewer. Adversarial inputs get red-teaming, with every successful attack added as a future test case.
Run the same quality system before and after release
Comparable measurements are the whole game. If the pre-release baseline and the production numbers are not measuring the same thing, you cannot tell when production drifts. Run the same evaluators on both sides, and feed every production failure back into the dataset.
Here is the five-step loop, checklist style:
Why continuous evaluation is the moat
A benchmark answers a question about a model someone else trained. An eval answers a question about a system you built. The teams that win stop treating evaluation as a pre-launch checkbox and run it as a loop: evals on their own data, in CI and in production, with a human in the loop, feeding every failure back as the next test.
That loop is also why fixed-flowchart tools do not cover this ground. Zapier, Make, and n8n run deterministic flows where there is one right path to test, so a pass/fail check on the final output mostly suffices. An agent picks tools and reasons, which means you have to evaluate the trajectory, not just the answer. And a chatbot only needs its single-turn answers checked, the easy 20% of the problem. The hard 80% is multi-step behavior, tool calls, and state. Our AI agents vs chatbots guide covers that gap in full.
Evaluation is the difference between shipping a model and running a system. Pick your model with benchmarks. Ship your system with evals. And keep the loop running where the work actually happens.

AI Agent Observability: Monitor and Debug in Real Time
Most AI agents fail not because they were built wrong, but because no one could see what was happening. Here is how AI agent observability helps you monitor, trace, and debug in real time before a silent failure becomes a production incident.
Frequently asked questions
What is LLM evaluation?
LLM evaluation is the practice of measuring whether a model, or an application built on one, works as intended. It runs at two layers: benchmarks score a model's raw ability on public tests, while evals score your system on your own data and failure modes. Together they tell you which model to pick and whether your application is good enough to ship.
How is LLM evaluation different from benchmarking?
Benchmarks are public test suites that score a model's raw capability in isolation, like MMLU or GSM8K. They help you pick a starting model but cannot predict production, because of saturation, contamination, and isolation from your prompts, tools, and users. Evals are tests you write against your own data, and only they decide whether your system works.
What is LLM-as-a-judge?
LLM-as-a-judge uses a language model with an evaluation prompt and a rubric to grade another model's output. It scales quality review beyond what humans or code can cover, reaching over 80% agreement with human evaluators on MT-Bench. A judge must be calibrated against human labels, because judges carry position, verbosity, and self-enhancement biases.
Do I need human labels to evaluate an LLM?
Not for every output, but yes for calibration. Human labels are the gold standard and the source of the golden dataset that calibrates the other methods, especially LLM-as-a-judge. Most teams mix human review on a small labeled set with code checks and judges on the rest.
How often should LLM evals run?
As a loop, not a one-off. Run smoke evals on every pull request, a full suite nightly or weekly, and online evals continuously on sampled production traffic. Every production failure should be added back to the dataset as a regression test so the loop keeps improving.
What is the difference between model evaluation and application evaluation?
Model evaluation asks whether the model works, measuring raw capability on shared benchmarks. Application evaluation asks whether the system built around it works, covering prompts, retrieval, tools, guardrails, latency, cost, and task completion. A strong model can still power a weak application, so both are needed.
How do you evaluate an AI agent, not just a single answer?
Score the trajectory, not just the output. Check whether the agent selected the right tool, extracted the right parameters, used the tool result correctly, recovered from failures, and reached the expected final state. A final-answer judge misses failures like a hallucinated tool call that happens to produce a lucky result.






