A robots.txt that blocks nothing by accident

robots.txt is a text file at the domain root that tells crawlers which URLs not to request, and it is not a way to remove pages from results. The distinction looks academic until a wrong group closes the whole site.

The syntax that matters

A group is a block of lines that starts with User-agent and continues with Allow, Disallow, Crawl-delay or non-standard directives. The crawler applies the most specific group that names it. When that group exists, the * rules do not add up: they are replaced. That is why every named group has to repeat the Disallow lines, or it opens what the generic group had closed.

User-agent: GPTBot
Allow: /
Disallow: /api/
Disallow: /preview
Disallow: /builder/

User-agent: *
Allow: /
Disallow: /api/
Disallow: /preview
Disallow: /builder/

Sitemap: https://ui.eneawork.it/sitemap.xml

Google recognises * and $ in paths and compares the longest prefix, with Allow winning when lengths are equal. Other engines implement a subset: an exotic rule can be ignored, and the file is public, so it holds no secrets. Google's documentation on the format, read on 1 November 2026, is the source for these rules.

The test that prevents a full block

cat > robots.mjs  /^User-agent:/m.test(g));
const sitemaps = txt.match(/^Sitemap:\s*\S+/gm) || [];
for (const g of groups) {
  const agent = (g.match(/^User-agent:\s*(.+)$/m) || [])[1];
  const closed = (g.match(/^Disallow:\s*\S+/gm) || []).length;
  console.log(agent, `${closed} disallow`, closed < 3 ? "check" : "ok");
}
console.log(`${groups.length} groups, ${sitemaps.length} sitemap lines`);
EOF
node robots.mjs

The script splits the file into groups, counts the Disallow lines and flags with check any group holding fewer than three. On the EneUI site it prints 14 groups, all with three Disallow lines, and one Sitemap line. Run the check after every edit: a half-copied group is the most common way to reopen /api/ without noticing.

The EneUI catalogue

In our file the content pages stay open: the 111,985 records under /components, the 1,556 blocks under /blocks and the 12,808 icons under /icons. /api/ is closed, with an exception for /api/registry/ for answer crawlers such as OAI-SearchBot and PerplexityBot. The rule is deliberate: the page gets crawled, the endpoint that burns CPU does not. Checked on 1 November 2026.

The limitation

A Disallow does not remove an already indexed URL: Google can keep showing the page with the anchor text of external links. Leaving the index needs noindex, which in turn needs crawl access: blocking the page in robots.txt stops Google from reading the noindex and leaves it in limbo. The file is cached too, so a fix is not visible instantly.

Where to continue

Automatic generation of the file in the App Router is in generated sitemaps and robots in Next.js; the sequence for getting indexed is in getting a site indexed by Google; the difference between noindex and canonical is in canonical, the practical guide.