Classifying support tickets with a small model

Classifying tickets is a routing decision: fixed labels, a confidence threshold and a human queue for whatever stays ambiguous.

Labels before the model

• Few and stable: five to twelve labels cover most tickets, each with an owner.
• One extra label: "other" is a valid label and must be counted, not hidden.
• Examples per label: ten already-routed tickets are worth more than a long description.

import { generateText } from 'ai';

const labels = ['billing', 'bug', 'shipping', 'account', 'other'] as const;

const result = await generateText({
  model: 'openai/gpt-5-nano',
  system:
    'Return one label from this list and nothing else: ' +
    labels.join(', ') +
    '. If none fits, return other.',
  prompt: ticket.subject + '\n\n' + ticket.body,
  maxOutputTokens: 8,
});

const label = result.text.trim().toLowerCase();
console.log(labels.some((l) => l === label) ? label : 'other');
console.log(result.usage);

Eight output tokens are enough for one word. If the model writes a sentence, the validator drops it and the ticket goes to the human queue: that is the correct behaviour, not an error to hide.

The threshold, not the label

A classification without confidence cannot be routed. Ask for a score from 0 to 1 as well, and send everything below 0.8 to a person: it costs less than one wrong billing ticket.

Batch pays half

A ticket is not as urgent as a chat: batch mode halves the price and accepts latency. With gpt-5-nano input drops to $0.025 per million, so the saving comes from volume, not from switching model.

The limitation

Quality depends on the examples, and this site has no labelled ticket set: no accuracy figure here is measured. A model trained on wrong labels learns the mistake as efficiently as it learns everything else.

Where to continue

How to evaluate a classifier is in evaluating an agent, instructions that hold up in prompt engineering for agents, cost per job in cost per task. The records: 111,985 components, 1,556 blocks, the 21st.dev sidebar.