Evaluating an AI feature: cases, thresholds and regressions in CI

Evaluating an AI feature means fixing a set of cases, a threshold and a test that runs on every change. Without those three, “it looks better” is the only measure you have.

The case set

• Between 20 and 50 real cases: real questions, not sentences invented to make the prompt pass.
• One case, one expected outcome: text that must appear, JSON that must validate, a refusal you expect.
• One hard case per kind: empty question, off-topic request, very long input.

From ticket to case

A set is built from the bottom up. Take the last 200 support tickets, group them by intent and keep the 5 most frequent intents; from each group pull 4 to 10 real questions, with the answer support actually gave. Forty cases come from here, not from the imagination of whoever writes the prompt. The user's text stays as it is, typos included, because that is what the model receives in production.

Every case carries four measures: task outcome (right or wrong, no half marks), correct refusal on off-topic requests, p95 latency and cost per call in tokens. None of the four is a single score: read them in a column, and open the case that got worse in full. The failure mode to watch is the right answer to the wrong question: plausible text, different intent, and the test still passes because it only looks for the expected word.

A test, not a benchmark

import { describe, expect, it } from 'vitest';

const CASES = [
  { input: 'dove trovo il pulsante di login?', expected: 'login' },
  { input: 'quanto costa il piano pro?', expected: 'piano' },
];

describe('assistant', () => {
  for (const c of CASES) {
    it(c.input, async () => {
      const answer = await ask(c.input);
      expect(answer.text).toContain(c.expected);
      expect(answer.usage.totalTokens).toBeLessThan(4000);
    });
  }
});

The test runs in CI like the others: token cost and latency are part of the outcome, not a separate report. If a case is unstable, note the reason instead of deleting it.

The thresholds

Write a threshold before you see the results: 90% of cases with the expected text, zero invalid JSON, a correct refusal on every off-topic request. A number chosen after the measurement is not a threshold, it is a description of what happened.

Regressions in CI

npx vitest run evals
npx promptfoo eval --config evals/promptfooconfig.yaml

The first command covers the deterministic cases, the second compares two models or two prompts on the same set. A model change that makes a case worse blocks the merge, like a broken test.

The limitation

A set of 40 cases does not cover the long tail of real questions, and a model can pass it by remembering the answers. Automatic evaluation measures what you wrote, not what the user means; for that you need a sample of real conversations. No single score replaces reading the failed cases.

Where to continue

The method for comparing two models is in eval and benchmark for an agent, handling unstable tests in finding flaky tests, the measured comparison between models for code in evaluating a model for code. To show the outcomes you need an interactive log table or a project data table, inside table components and 111,985 components.