Robots.txt Configuration for Multi-Locale Headless Sites

A robots.txt baked at build time freezes crawler directives to one environment and one routing table. Copied into a preview deployment, it invites crawlers into staging; copied to a new country domain, it points at the wrong sitemap. This guide generates robots.txt per host and per environment from the same configuration that drives locales and sitemaps. It’s part of Dynamic Sitemap Generation, within Localization & SEO Optimization.

Why Build-Time Generation Fails

Most CI/CD workflows compile robots.txt as a static asset, which locks directives to one deployment. The same file ends up on production, on preview deployments and on staging hosts, although they need opposite rules. Sites with country domains, such as example.de and example.fr, need a file per host whose Sitemap line points at that host’s sitemap. Decouple the file from static assets and generate it per request in a route handler, where the host and environment are known.

Two facts shape everything else. A robots.txt file applies only to the host that serves it, at exactly /robots.txt; there is no per-path or per-locale robots file, so subfolder locales share their host’s file. And robots.txt controls crawling, not indexing: disallowing a locale prevents crawlers from seeing its canonical and hreflang tags, which usually makes international SEO worse, not better.

Runtime Architecture

Query the CMS locale registry at request time for active locales, fallback chains, and environment flags, then build the file in a route handler. This keeps crawler directives aligned with the routing table and lets you block preview branches conditionally — no manual file swaps, no environment-specific build steps.

Next.js App Router Implementation

Add a dynamic route handler at app/robots.txt/route.ts. It fetches locale config from the CMS API or env vars and returns a text/plain response with strict cache headers, so the edge can’t serve stale directives.

import { NextResponse } from 'next/server';
import { getActiveLocales } from '@/lib/cms-locale-registry';

// Force dynamic rendering to bypass static generation
export const dynamic = 'force-dynamic';

export async function GET() {
  // Fetch live locale registry from CMS or configuration store
  const locales = await getActiveLocales();
  // NODE_ENV is "production" on preview deployments too; use the deployment environment instead.
  const isProduction = process.env.DEPLOY_ENV === 'production';
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://yourdomain.com';

  const directives = isProduction
    ? [
        `User-agent: *`,
        `Disallow: /api/`,
        `Disallow: /search`,
        `Disallow: /*?*sort=`,
        ``,
        // One index referencing every locale's chunks; per-locale sitemaps can be listed too.
        `Sitemap: ${baseUrl}/sitemap-index.xml`,
        ...locales.map((locale) => `Sitemap: ${baseUrl}/${locale}/sitemap.xml`),
      ]
    : [`User-agent: *`, `Disallow: /`];

  return new NextResponse(directives.join('\n') + '\n', {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      // Short edge cache: changes propagate within minutes without hammering the origin.
      'Cache-Control': 'public, max-age=300, s-maxage=300',
    },
  });
}

Abstracting locale resolution into @/lib/cms-locale-registry keeps the handler environment-agnostic. force-dynamic stops the framework from prerendering the file into every deployment, and the short cache lifetime lets rule changes propagate within minutes. Crawlers cache robots.txt themselves for up to a day, so never rely on instant changes.

Structures: Subfolders, Subdomains and Country Domains

How many robots files a multi-locale site needs depends on its URL structure. With subfolders (example.com/de/), there is one host and one robots.txt for all locales; rules cannot differ by locale except through path patterns, and should rarely need to. With subdomains (de.example.com), each subdomain is a host with its own file, generated by the same handler from the host name. With country domains (example.de), the same applies, and each file’s Sitemap line must point at a sitemap on that domain, unless you have verified cross-domain submission in each search engine’s tools.

What to Disallow, and What Not To

Disallow paths that waste crawl budget without ever deserving to rank: internal API routes, on-site search results, cart and account pages, and parameter variants such as sort orders that create near-infinite URL combinations. Do not disallow locales, fallback pages or preview-looking paths on production to hide duplicates; crawlers then cannot read the canonical, hreflang and noindex signals that actually resolve duplication, as described in canonicalizing fallback pages. A disallowed URL can still appear in search results, without a description, if other sites link to it. To keep a page out of results, allow crawling and use noindex.

Preview and Staging Hosts

Every non-production host must serve Disallow: /, and ideally also require authentication, since robots.txt is a request that polite crawlers honour, not an access control. Deciding by deployment environment rather than host name lets new preview hosts inherit the right rules automatically. Add an X-Robots-Tag: noindex header to every response on non-production hosts as a second line of defence, which covers pages that crawlers reach despite the disallow, for example through links shared publicly.

Testing the Output

Because one wrong line can remove a site from search, test robots.txt like critical code. A unit test calls the handler for each production host and asserts the exact expected output: the right disallow lines, no Disallow: /, and a Sitemap line per host with an absolute URL on that host. Another test calls it with a preview environment and asserts disallow-all with no sitemap. In the post-deploy audit, fetch /robots.txt from every production host and compare it with the expected output, and fetch a few listed sitemap URLs to confirm they return 200 and valid XML. Finally, check that no URL in any sitemap is blocked by the file, using a robots parser library, since listing blocked URLs is a contradiction search engines report as an error.

// robots.test.ts
import { GET } from "@/app/robots.txt/route";

test("production robots allows crawling and lists the sitemap", async () => {
  process.env.DEPLOY_ENV = "production";
  process.env.NEXT_PUBLIC_SITE_URL = "https://www.example.com";
  const body = await (await GET()).text();
  expect(body).not.toMatch(/^Disallow: \/$/m);
  expect(body).toMatch(/^Sitemap: https:\/\/www\.example\.com\/sitemap-index\.xml$/m);
});

test("preview robots blocks everything", async () => {
  process.env.DEPLOY_ENV = "preview";
  const body = await (await GET()).text();
  expect(body).toMatch(/^Disallow: \/$/m);
  expect(body).not.toMatch(/Sitemap:/);
});

Gotchas & Edge Cases

• Environment detection. NODE_ENV is production in every optimized build, including previews. Use a variable that distinguishes production deployments, which most hosting platforms provide.
• One wrong line. A leftover Disallow: / from staging deployed to production removes the site from crawling within days. Test the production file in CI.
• Sitemap URLs must be absolute. Relative Sitemap lines are ignored.
• Caching too long. A robots.txt cached for a week at the CDN delays fixes. Keep edge caching short.

Worked Example

A company with country domains for six markets served one static robots.txt from its build, with a single Sitemap line pointing at the .com sitemap. None of the country domains’ sitemaps were referenced, and preview deployments served the production file, so crawlers had found and indexed a few hundred preview URLs. A per-host handler with environment detection fixed both: every country domain now references its own sitemap index, previews serve a disallow-all file and a noindex header, and a CI test asserts the production output for each host. The indexed preview URLs disappeared within a month.

Rollout Checklist

• Generate robots.txt per request from host and deployment environment.
• Serve disallow-all plus a noindex header on every non-production host.
• Reference each host’s own sitemap index with absolute URLs.
• Disallow only crawl traps such as search results and parameter variants.
• Never disallow locales or fallback pages to handle duplicates.
• Test the production output for every host in CI.

Frequently Asked Questions

Related

• Dynamic Sitemap Generation
• Automated SEO Audits for Headless CMS Deployments
• Handling Canonical URLs in Headless Multilingual Setups