AI & MCP
Evaluating a model for code: fixtures, gates and cost per successful task
A coding model is chosen with your own fixture, a gate that passes or fails and the cost per successful task. Public benchmarks are not enough.
Key points
- The useful metric is cost per successful task, not an average score on a public benchmark.
- The fixture must come from your repository and have a runnable gate: build, types, tests.
- The first-attempt rate matters too, because every retry multiplies input tokens.
A coding model is evaluated with a fixture taken from your own repository, a gate that passes or fails and the cost per successful task. An average score on a public benchmark answers a different question.
The metric that survives a model swap
The useful metric is a fraction with two real numbers: total cost divided by successful tasks. Total cost comes from the API response, in the usage field; successful tasks are decided by your gate, not by the model claiming it finished. When you change vendor the fraction stays comparable even as tokens per task change.
Building the fixture in an afternoon
- 20 tasks taken from real, already closed bugs: you have the expected diff and nobody has to invent it.
- One gate per task:
npm run build,tsc --noEmitor a test that failed before and passes after. - A time limit and a limit on files touched, otherwise an agent rewrites the repository to make one test pass.
- The bare prompt: no hints in the text, because a good prompt hides a weak model.
The runnable gate
npm run build && npx tsc --noEmit -p client/tsconfig.jsonIn the EneUI repository npm run build runs the client build, a syntax check and CSS minification. A gate like that does not ask the model to grade itself: either the process exits with 0 or the task failed. Run it on a clean copy, because a build that finds stale artefacts can pass for the wrong reason.
The three columns of the sheet
| Column | How you get it | Why it matters |
|---|---|---|
| Successful tasks | gate with exit code 0 | filters out answers that look right and break |
| First-attempt rate | retries declared in the log | every retry resends the context and doubles the tokens |
| Cost per successful task | sum of usage divided by successes | compares models with different prices and verbosity |
| Files touched | diff before and after | a fix that changes 40 files is tomorrow's regression |
Price and tokens are two numbers
A model that is cheap per token can cost more per task if it writes three times as much or misses on the first attempt. An expensive model can win by closing first time in a large repository. The gap between the two list pages read on 1 November 2026 is two orders of magnitude on input, so the success rate has to be measured, not assumed.
The limitation
Our fixture measures 60 files and 13,924 lines: a small project, and a model that shines here can get lost in a monorepo of 400 packages. Public benchmarks have the opposite problem, because their tasks have often been seen during training. Neither number predicts your case, so rebuild the fixture when the repository changes shape.
Where to continue
Token cost in a long context is in long context: real costs, and the order of steps in orchestrating multi-step agents. For the risks of an agent writing to disk read securing a coding agent, and to choose among ready-made interfaces the review of AI UI generators. The fixture can start from the catalogue: 111,985 components, 1,556 blocks and a record with a declared MIT licence such as dqnamo agent dock.
Frequently asked questions
How many tasks does a credible fixture need?
Twenty are enough to see large differences, meaning a model that closes on the first try and one that never closes. Fewer than ten lets noise hide the signal, more than fifty costs more to prepare than it returns.
Can I use a public benchmark as a first filter?
Yes, to rule out models that cannot write code at all. It cannot choose among the top five, because public tasks are known and the prompt format differs from yours. The fixture decides, the benchmark orients.
How do I count tokens without depending on the vendor?
Read the usage field of every response and add it up: that is the only number you actually pay. The four-characters-per-token estimate is good for a quote, never for a comparison.
Is the first-attempt rate worth more than cost?
It depends on human time. An automatic retry costs tokens, a retry you re-read costs attention too. If your bottleneck is review, measure reviews per task rather than tokens.
What makes a fixture useless?
A gate the model can edit. If the agent can change the test or the tsconfig, the result is a claim, not a measurement. The gate must live outside the folder the agent writes to.