Automatic comment moderation: rules, a model and an appeal

Automatic comment moderation is a chain of three filters: deterministic rules, a model for context and a human queue for doubtful cases. Every decision must be recorded with its reason, or it cannot be contested.

Three layers

• Rules: links, capitals, repeated characters, a word list. Almost no cost and a decision you can explain.
• Model: reads the whole sentence and tells an insult from a quotation; it only decides in the doubtful band.
• Human queue: doubtful cases go to a person, with the text and the reasons attached.

The rules, in code

const RULES = [
  { name: 'links', test: (t) => (t.match(/https?:\/\//g) || []).length > 1 },
  { name: 'caps', test: (t) => t.length > 40 && t === t.toUpperCase() },
  { name: 'repeat', test: (t) => /(.)\1{6,}/.test(t) },
];

export function screen(text) {
  const hit = RULES.find((rule) => rule.test(text));
  return hit
    ? { action: 'review', reason: hit.name }
    : { action: 'allow', reason: 'rules' };
}

The function is synchronous and does not call the network: it runs inside the request that saves the comment. When a rule fires, the comment is not published at once but goes to review; no rule deletes anything. Try the file with node --test moderation.test.js.

The model decides only the doubt

The model gets one question: “does this comment attack a person or a group?”. The answer is a value with a confidence, and you choose the threshold on your own data. Below the low threshold it publishes, above the high one it hides, in between a person decides.

What to record

• Text and author, with the date, so the case can be rebuilt.
• Reason: the rule name or the model label, not “spam”.
• Version: model name and version, so a regression can be explained.
• Outcome: published, hidden or queued, with who decided.

The appeal

Whoever wrote the comment must be able to ask for a review without opening a separate ticket: a link in the notification is enough. The reviewer sees the text, the reason and the previous decision, and can overturn it. A system without an appeal shifts the harm from spam to error.

The limitation

Rules do not read context: a word list hits a quotation and an insult with the same weight, and obfuscation with substituted characters gets through. A model adds cost and latency to every comment and needs a fresh review when its version changes. Neither reads sarcasm better than a person.

Where to continue

Input sanitising rules are in sanitising input, the duty to check on the server in server-side validation, request limits in rate limit design. For the review queue you need a table with status or a project data table, inside table components and 111,985 components.