Evaluating an agent with a benchmark: harness, pass^k and variance

An agent benchmark has four parts: fixed tasks, an environment that resets, an automatic grader and k repetitions per task. Without the fourth number the result is an anecdote.

The four parts of a harness

• Task: a request with an expected result and a starting context identical for every model.
• Environment: a clean copy of the repository, without node_modules or dist left over from a previous round.
• Grader: a process that decides pass or fail without asking the model how it went.
• Repetitions: at least three rounds per task, because an agent does not answer twice the same way.

The grader is the part most often got wrong. A test the agent can edit is not a grader, it is a suggestion: run it from outside, on files the agent cannot touch. The same goes for the time limit and the maximum number of changed files, which are decided before seeing the result.

pass@k and pass^k

pass@k asks whether at least one of k attempts succeeds, and it is the right metric for capability: an agent that never gets there is out. pass^k asks whether all k attempts succeed, and it is the right metric for reliability: an agent that succeeds once in three tries is not ready for an automated flow.

node -e "const p=0.8;for(const k of [1,3,5])console.log('pass^'+k,(100*p**k).toFixed(0)+'%')"

With an 80% success probability the command prints pass^1 80%, pass^3 51% and pass^5 33%. The same agent looks excellent on one attempt and mediocre on five: the difference is not in the model, it is in the question the benchmark asks.

Variance with 20 tasks

node -e "const n=20,p=0.8,se=Math.sqrt(p*(1-p)/n);console.log((100*p).toFixed(0)+'% +/- '+(196*se).toFixed(0)+' pt')"

The command prints 80% +/- 18 pt: with twenty tasks a measured 80% is worth anything from 62% to 98%. Two models ten points apart on twenty tasks are indistinguishable. Getting down to a five-point margin takes hundreds of tasks, or fewer tasks with a grader finer than pass or fail.

Three graders, three truths

For a coding agent the first is mandatory, the second helps on small tasks, the third only where no oracle exists. An automatic judge must be validated against a hand-graded sample: if it disagrees on twenty cases, the number it produces is not a measurement.

Contamination and noise

A public task has almost always been seen in training, so the score measures the model's memory rather than its capability. The harness adds noise too: prompts unlike the real ones, a warm cache on the first round, tools answering at different latencies. The defence is boring: tasks taken from closed bugs in your repository, an environment rebuilt every round, the same prompt for everyone, declared repetitions.

What to report

• Task count and repetitions, because without those two numbers the rest is unreadable.
• Success with an interval, not the point: 80% on twenty tasks is 80% ± 18 points.
• Cost per successful task, which folds tokens and failures into one number.
• Median steps and files touched, which predict future maintenance better than the score.

The limitation

Our harness runs two tasks: it checks that a model produces valid JSON and coherent JSX, it does not rank models. A large harness is blind in the same way: an agent can pass every task and still be unusable because it is slow, expensive or unsafe on disk. The score is a filter, the decision stays human.

Where to continue

Building the fixture and the cost per successful task are in evaluating a model for code, the order of steps in orchestrating multi-step agents, the risks of writing to disk in securing a coding agent. The catalogue behind the tasks is 111,985 components and 1,556 blocks; a citable record is dqnamo agent dock.