Evals for AI agents: separate the maker from the verifier

In traditional software, nobody approves their own pull request. There are automated tests, peer review and a pipeline that blocks the deployment if something breaks. With AI agents, many organizations drop that discipline precisely when they need it most: the same model generates the answer, decides it is fine and executes the action. It works in the demo and fails in production in ways that are hard to spot.

The rule we propose is simple to state and demanding to implement: the one who verifies is not the one who does the work. This is the independent verification layer, one of the seven in an agent harness, and probably the one that most separates a flashy pilot from a system a regulated organization can trust. This article explains why an agent can't evaluate itself, what types of evals exist, how to build them into your development cycle and how to put an inline verifier in front of every action that has real effects.

Why an agent can't grade itself

The instinct to "ask it to check its answer" is reasonable, but the evidence doesn't back it up. The paper Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024) found that models struggle to correct their reasoning without external feedback, and that their performance sometimes degrades after trying. When the model that made the mistake reviews its work, it brings the same blind spots that caused the error.

There is a second problem: when a model acts as a judge, it tends to favor itself. The study Judging LLM-as-a-Judge documented position, verbosity and self-enhancement biases in LLM judges (preferring answers generated by the model itself). An agent that grades itself combines both flaws.

What does work is introducing an independent signal. Anthropic's guide Building effective agents stresses that agents need to get "ground truth" from the environment at each step (tool results, code execution) and describes the evaluator-optimizer pattern, in which one call generates and another evaluates and provides feedback in a loop. The key is that the evaluator has information or criteria the generator doesn't.

Types of evals

No type of eval is better than the rest; each one fits a different question. Anthropic's guide Demystifying evals for AI agents groups them into code-based, model-based and human graders, each with different strengths and costs. In practice, it pays to split the first group in two:

Type	What it verifies	Strengths	Weaknesses	Use it for
Deterministic	Exact outcome or system state: the record exists, the test passes, the JSON validates against the schema	Fast, cheap, reproducible	Rigid in the face of valid variations	Tool calls, formats, effects on systems
Rule-based	Business conditions: amounts under the threshold, required fields, no prohibited data	Explainable, auditable	Only covers what someone anticipated	Policies, compliance, guardrails
LLM-as-judge with a rubric	Open-ended quality: tone, completeness, faithfulness to the source	Flexible, scalable, captures nuance	Non-deterministic, more expensive, needs calibration	Writing, summaries, customer responses
Human	Expert business judgment	The reference standard	Slow and expensive	Calibrating the others, ambiguous cases, sample-based audits

Three principles that change the quality of your evals:

Grade the outcome, not the path. As the same guide points out, it is usually better to evaluate what the agent produced rather than the route it took, so you don't penalize valid alternative solutions. If the support agent was supposed to issue a refund, verify that the refund exists in the system with the correct amount, not that it called the tools in the order you imagined.
An LLM judge needs a rubric and calibration. The rubric must be concrete ("does it cite the case number? does it promise timelines that aren't in the policy?"), not "rate the quality from 1 to 10". And its scores must be compared periodically with those of human experts; if they diverge, the judge is misconfigured.
Keep judge and generator separate. A different model, a different provider, or at least a different prompt with different information. If you use the same model with the same context, you're back to self-evaluation.

Golden sets: your executable specification

A golden set is a collection of representative tasks with their expected outcomes, validated by the people who know the business. In practice, it is the agent's specification written so that a machine can check it.

How to build one without turning it into an endless project:

Start with real failures. Anthropic suggests starting with 20 to 50 simple tasks drawn from real failures. That is enough to catch regressions, and it forces you to look at what actually goes wrong.
Cover the categories, not just the happy path. Ambiguous requests, missing data, manipulation attempts in the input text, cases that must be escalated, cases where the right answer is "I can't do that".
Keep the full trace. Input, retrieved context, tool calls, output. When a test fails, you need to see where it went off track.
Every production incident adds a case. The set grows with use, and it becomes much less likely that you'll trip over the same stone twice without noticing.
It has an owner and a version. If nobody from the business reviews it, it falls out of step with policy and starts rewarding answers that are no longer correct.

Regression evals in CI: nothing ships without passing here

In an agent, "the code" includes the system prompt, the tool descriptions, the context retrieval configuration and the model itself. Changing any of them can break behavior that used to work. That is why evals must run in your continuous integration pipeline, just like unit tests.

Anthropic's guide distinguishes two suites with different purposes: capability evals, which start with a low pass rate because they target what the agent can't do well yet, and regression evals, which should stay close to 100% and protect against backsliding. Mixing them creates noise: you can't tell whether the score dropped because things got worse or because you added hard cases.

Details that make the difference:

Run each task several times. Agents are not deterministic. Anthropic's guide distinguishes pass@k (the probability of at least one success in k attempts) from pass^k (the probability that all k attempts succeed). For a customer-facing agent, what matters is consistency: pass^k.
Explicit thresholds per category. A change that improves the writing but lowers tool accuracy should not pass. Define which metrics block and which are informational only.
Read the traces. It is a recommendation the Anthropic guide on evals keeps coming back to: review the transcripts of failed tests and of a sample of the passing ones. That's how you find out the grader is rewarding the wrong thing.
A model change is a full release. Upgrading the model version, or switching providers, is the change most likely to cause a silent regression. Treat it that way.

Inline verification: a verifier before every action with real effects

Offline evals tell you whether the agent is good on average. They don't protect you from the specific wrong action it is about to take right now. For that you need inline verification: an independent component that reviews the proposed action before it touches a real system.

How to design the verifier:

Order checks from cheapest to most expensive. Deterministic rules first (does the amount exceed the authorized threshold? does the customer exist? is the format valid?). Only if they pass, an LLM judge for what the rules don't cover. That way most rejections cost milliseconds.
Validate against the source, not against the agent's story. If the agent says the invoice is a duplicate, the verifier queries the ERP. The verifier doesn't trust the executor's reasoning; it trusts the systems.
Authorization lives outside the model. In its LLM06:2025 Excessive Agency risk, OWASP recommends enforcing authorization in downstream systems instead of letting the LLM decide whether an action is allowed, and requiring human approval for high-impact actions.
Rejections come with a reason. A retry only helps if the executor receives a concrete reason ("the amount exceeds the cap authorized by policy"). A bare "rejected" produces the same mistake in different words.
Cap the retries. After N rejections, or if the action is high-impact, the case goes to a person. We go deeper into that design in guardrails and human oversight.

Not every action deserves the same rigor. Reading a catalog doesn't need a verifier; issuing a payment does. Classify your tools by impact and reversibility, and apply proportional verification.

Metrics that matter

Many agent metrics measure activity, not outcomes. These three give an honest reading and are understandable outside the technical team:

Metric	What it measures	How it's calculated	Warning sign
Task success rate	Whether the agent resolved the case end to end	Tasks with a verified correct outcome / total tasks	Volume goes up but the success rate doesn't
Tool accuracy	Whether it called the right tool with the right parameters	Valid and necessary calls / total calls	Many redundant calls or calls rejected by the verifier
Human escalation rate	How much human work it still needs	Escalated cases / total cases	Drops sharply without the success rate going up (the agent stopped escalating what it should)

Add the verifier rejection rate as an early indicator: if it rises, something changed in the inputs, the systems or the model before a customer notices. These metrics are the foundation of agent operations and governance and fit the Measure function of the NIST AI RMF.

Common mistakes

Evaluating only the happy path. The test set was written by the person who designed the agent, using the examples they know work.
A judge without a rubric. "Is this a good answer?" produces scores that mean nothing and change from run to run.
The same model as judge and executor, with the same context. That's self-evaluation with extra steps.
Measuring a single run. As an illustrative example: 90% on one run can mean 60% real consistency.
Evals nobody reads. The pipeline stays green for months because the grader approves everything. Review the traces.
A frozen golden set. Policies change; if the expected answers don't, the agent learns to follow outdated rules.
Verifying the text, not the effect. The agent says "refund issued", but nobody checks that the refund exists in the system.
No retry cap. A maker-verifier loop without a limit turns into an expensive cycle that never converges.

How we approach it

At Nivelics, verification is part of custom AI agent engineering from the first sprint: we define the golden set with the business, separate executor and verifier, connect the evals to the CI pipeline and keep the metrics visible to the people accountable for the agent. We work with Claude, models on Amazon Bedrock or Azure OpenAI and open models, and orchestrate with tools such as LangGraph or the Vercel AI SDK depending on the case.

If you have an agent that works in the demo but you don't dare let it act on its own, let's talk: we start by measuring where it fails today.

Frequently asked questions

Why isn't it enough to ask the agent to review its own answer?

Because it shares the same blind spots that led to the mistake. Research on self-correction shows that, without external feedback, models struggle to correct their reasoning and sometimes get worse after trying. Useful verification needs an independent signal: a test, a rule, another evaluator or a person.

What is a golden set?

A collection of representative tasks with their expected outcomes, reviewed by business experts, against which the agent is measured every time something changes. Start small, with cases taken from real failures, and grow it with every production incident.

Can you trust an LLM as a judge?

Yes, with conditions: an explicit rubric, a model or configuration different from the one that generated the answer, and periodic calibration against human evaluators. LLM judges have documented biases, such as favoring the first option presented, longer answers or answers they generated themselves.

When should a human step in?

When the action has a high or irreversible impact (payments, contract changes, sensitive external communications), when the verifier rejects several times in a row or when confidence is low. OWASP recommends requiring human approval for high-impact actions in LLM-based systems.

Which metrics should an executive committee see?

Three are enough to start: end-to-end task success rate, tool-use accuracy and human escalation rate. Together they show whether the agent gets the job done, whether it does so safely and how much human work it still needs.