Dynamic Sitemap Generation

A static XML sitemap goes stale the moment content scales across nested taxonomies, locales, and draft states — in a headless stack the sitemap is a build artifact, not a file. This guide treats it as a data pipeline: a status-filtered CMS query, framework-native route enumeration, and edge caching that keeps the sitemap fresh without rebuilding the site. The goal is to feed crawlers only canonical, published URLs. It’s a building block of Localization & SEO Optimization; the implementation lives in Generating XML sitemaps from headless CMS routes.

Integration Contract

A sitemap is a promise to search engines that every listed URL is canonical, indexable and returns 200. Keeping that promise requires the sitemap generator to share its decisions with the rest of the site. URLs: built with the same function the router and canonical tags use, from the same slug and locale fields, as absolute URLs on the canonical host. Inclusion: derived from the same flags as robots meta tags and canonicals, so a page marked noindex or falling back to another locale never appears. Alternates: hreflang entries listing only locales where the page genuinely exists. Freshness: lastmod from the content’s real modification time, updated when a publish changes something visible, not on every rebuild.

# .env: sitemap generation
SITE_ORIGIN=https://www.example.com
SITEMAP_LOCALES=en,de,fr,ja
SITEMAP_CHUNK_SIZE=10000
SITEMAP_CACHE_SECONDS=3600

Route Discovery & CMS Queries

The sitemap moves through four stages — a status-filtered query, route enumeration, edge caching, and post-build validation — each feeding the next.

Start with a deterministic fetch of every routable entity. Contentful, Sanity, and Strapi expose GraphQL or REST endpoints built for bulk retrieval. Filter strictly by publication status, locale, and slug; exclude drafts and archived entries unless you’re targeting a preview environment. Request only slug, updatedAt, locale, and changefreq via projection queries, and flatten nested structures at the query layer to avoid expensive client-side recursion.

*[_type in ["post", "page", "category"] && defined(slug.current) && status == "published"] {
  "slug": slug.current,
  "type": _type,
  "lastmod": _updatedAt,
  "locale": coalesce(locale, "default"),
  "priority": select(
    _type == "page" => 1.0,
    _type == "post" => 0.8,
    0.5
  )
}

Missing localized routes need Content Fallback & Routing so fallback URLs don’t pollute the index. Run queries against read-optimized CDN endpoints with retry logic for transient failures.

Framework Implementation

The framework dictates how the sitemap reaches crawlers. In the Next.js App Router, generateSitemaps() and generateStaticParams() enumerate routes programmatically: return locale identifiers, fetch per locale, and stream the XML response to avoid memory spikes on large builds.

// app/sitemap.ts
import { MetadataRoute } from 'next';
import { createClient } from '@sanity/client';

const client = createClient({
  projectId: process.env.SANITY_PROJECT_ID!,
  dataset: 'production',
  apiVersion: '2024-01-01',
  useCdn: true,
});

export async function generateSitemaps() {
  return [{ id: 'en' }, { id: 'es' }, { id: 'fr' }];
}

export default async function sitemap({ id }: { id: string }): Promise<MetadataRoute.Sitemap> {
  const query = `*[_type in ["post", "page"] && locale == $locale && defined(slug.current) && status == "published"] {
    "url": "/" + $locale + "/" + slug.current,
    "lastModified": _updatedAt
  }`;

  const routes = await client.fetch(query, { locale: id });

  return routes.map((route: { url: string; lastModified: string }) => ({
    url: `https://www.example.com${route.url}`, // sitemap URLs must be absolute
    lastModified: new Date(route.lastModified),
    changeFrequency: 'weekly',
    priority: 0.8,
  }));
}

Nuxt 3 uses server routes or @nuxtjs/sitemap; Astro uses getStaticPaths() plus community plugins. Whatever the framework, explicit locale routing prevents duplicate indexing — and Route Mapping for Multilingual Sites keeps hreflang annotations aligned with sitemap entries.

Getting lastmod right

lastmod is the one optional sitemap field search engines actually use, and only if it is trustworthy. Set it to the time the page’s visible content last changed, which is usually the entry’s publish time in the requested locale, not the time of the last build or the latest change to any referenced entry. If every URL shows today’s date after every deploy, search engines learn to ignore the field. For pages that aggregate other content, such as category listings, use the most recent publish time of the entries shown. Store per-locale publish times where the CMS supports them, since a German translation updated today should not bump the English page’s lastmod.

Regenerating on publish

For sites that publish often, regenerate only the affected chunk when content changes. A webhook handler maps the published entry to its content type and locales, revalidates those chunks and the index, and leaves everything else cached.

// app/api/sitemap-revalidate/route.ts
import { revalidateTag } from "next/cache";
import { verifyWebhook } from "@/lib/webhooks";

export async function POST(req: Request) {
  const raw = await req.text();
  if (!verifyWebhook(raw, req.headers.get("x-webhook-signature"))) return new Response("invalid", { status: 401 });
  const { contentType, locales } = JSON.parse(raw) as { contentType: string; locales: string[] };
  for (const locale of locales) revalidateTag(`sitemap:${locale}:${contentType}`);
  revalidateTag("sitemap:index"); // lastmod of the changed chunks appears in the index
  return Response.json({ revalidated: locales.length }, { status: 202 });
}

Each chunk’s data fetch is tagged sitemap:{locale}:{type}, and the index is tagged sitemap:index. The incremental regeneration guide covers chunk boundaries that stay stable as content grows.

Caching & Edge Delivery

Sitemap endpoints trade freshness against CDN efficiency. Use stale-while-revalidate with a conservative max-age (around 1 hour) to absorb traffic without serving stale data indefinitely. At scale, pre-generate sitemaps in CI/CD and persist them to object storage (S3, Cloudflare R2) to shed origin load — which pairs with Incremental sitemap regeneration for dynamic CMS routes so invalidation fires only when a content type actually changes.

Google’s sitemap guidelines cap a sitemap at 50MB uncompressed and 50,000 URLs. Past that, use a sitemap index (sitemap_index.xml) referencing locale- or type-chunked sitemaps at stable URLs. Crawlers fetch every chunk listed in the index, so each chunk’s URL must return the same content for every requester; never vary sitemaps by Accept-Language or other request headers.

Hreflang in the sitemap

Sitemaps can carry hreflang alternates for each URL with xhtml:link elements, which is often easier to keep consistent than hreflang tags in page heads, especially for large sites. Each URL entry lists every language version, including itself, and every listed version must in turn list the others. Generate them from one query that returns, for each page, the set of locales with real content, as described in canonicalizing fallback pages. Choose one place for hreflang, either the sitemap or the page head, or make sure both are generated from the same data; contradictions between them are a common source of ignored annotations.

Streaming large sitemaps

Chunks of ten thousand URLs with hreflang alternates can reach several megabytes of XML. Building them as one string in memory works on a server but can exceed memory limits in serverless and edge functions. Stream the XML instead: write the header, then each URL entry as the query cursor yields it, then the footer. Streaming also lets the response start before the whole query finishes, which reduces the chance of timeouts. Compress responses with gzip or Brotli at the CDN; crawlers accept compressed sitemaps, and XML compresses very well.

// app/sitemaps/[chunk]/route.ts: stream one chunk
import { iterateSitemapEntries } from "@/lib/cms/sitemap-source";

export async function GET(_: Request, { params }: { params: Promise<{ chunk: string }> }) {
  const { chunk } = await params; // e.g. "de-articles-1"
  const encoder = new TextEncoder();
  const stream = new ReadableStream({
    async start(controller) {
      controller.enqueue(encoder.encode('<?xml version="1.0" encoding="UTF-8"?>\n<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">\n'));
      for await (const e of iterateSitemapEntries(chunk)) {
        const alternates = e.alternates.map((a) => `<xhtml:link rel="alternate" hreflang="${a.locale}" href="${a.url}"/>`).join("");
        controller.enqueue(encoder.encode(`<url><loc>${e.url}</loc><lastmod>${e.lastmod}</lastmod>${alternates}</url>\n`));
      }
      controller.enqueue(encoder.encode("</urlset>\n"));
      controller.close();
    },
  });
  return new Response(stream, { headers: { "Content-Type": "application/xml; charset=utf-8", "Cache-Control": "public, s-maxage=3600, stale-while-revalidate=86400" } });
}

Escape URLs for XML, since & in query strings must become &; building URLs from slugs without query strings avoids the issue entirely.

Validation

Generating the sitemap is half the job; keeping it accurate needs continuous checks. Validate XML structure, URL accessibility, and lastmod formatting in the deploy pipeline, and lint for mixed HTTP/HTTPS, trailing-slash inconsistencies, and preview URLs leaking into production. Run Automated SEO audits for headless CMS deployments post-build to cross-reference sitemap URLs against live routes, validate robots.txt, and confirm canonical alignment. The Next.js sitemap file convention covers framework-level generation.

Preview & Draft Isolation

Sitemaps must never contain preview or staging URLs, and preview environments must never serve a sitemap that search engines could find. Generate sitemaps only with the delivery token, which cannot read drafts, and build URLs from the configured production origin rather than the request host, so a sitemap generated in a preview deployment still points at production and is harmless if fetched. Better still, return 404 for sitemap routes and a disallow-all robots.txt on preview hosts. Audit the production sitemap for any host other than the canonical one; a single staging URL in a sitemap can get a whole staging environment crawled.

Error Handling & Resilience

A sitemap generated during a CMS outage can be empty or truncated, and serving it tells search engines that thousands of pages disappeared. Treat generation failures as failures: if a query errors or returns far fewer URLs than the previous version, keep serving the previous sitemap and alert. A simple guard, refusing to publish a sitemap that shrank by more than a set percentage without an explicit override, prevents most accidents. Use cursor-based pagination for large collections, since offset pagination over changing data can skip or duplicate entries, and retry transient failures per page rather than restarting the whole run.

Testing Sitemap Generation

Unit-test the pieces that encode decisions: the URL builder with tricky slugs and locales, the inclusion rules per content type and flag, the hreflang alternates for pages translated into some locales but not others, and XML escaping. Then run an integration test against a staging CMS that generates each chunk and validates it against the sitemap schema. In CI, compare the new sitemap with the production one and report additions and removals by content type and locale; a large unexpected removal is exactly the kind of change the shrink guard exists for, and seeing it in a pull request is better than seeing it in search console. Finally, the audit job fetches a sample of listed URLs and checks that each returns 200, has a self-referencing canonical and no noindex tag.

Monitoring Indexing

A correct sitemap is only useful if search engines process it. Submit the sitemap index in each search engine’s webmaster tools and reference it in robots.txt. Review the coverage reports regularly: the number of submitted URLs, how many are indexed, and the reasons for exclusions. A large gap between submitted and indexed URLs for one locale or content type usually points to a quality or duplication issue, such as fallback pages that slipped into the sitemap or thin listing pages. Track the numbers over time per chunk; because chunks are split by locale and type, the reports show exactly where problems are. Pair this with log analysis of crawler requests, which shows how often crawlers fetch the sitemap and how quickly they visit newly listed URLs.

Sitemaps for Multi-Site Platforms

Platforms serving several sites or tenants from one codebase must generate a separate sitemap per host, listing only that host’s URLs. Resolve the tenant from the host, as with any other page, and never generate URLs for other hosts, since a sitemap may only list URLs on its own host unless cross-site submission is verified. Cache sitemaps per host and tag them with the tenant id, so one tenant’s publishes do not regenerate others’ sitemaps. The tenant-aware invalidation guide covers the tagging.

Worked Example

A travel publisher with 60,000 pages in four languages generated one sitemap at build time, listing every page in every locale. The build took longer each month, lastmod was always the build date, and a third of the listed URLs were English fallbacks on localized paths. The team split the sitemap into chunks by locale and content type, generated each on request with a one-hour cache and tag-based revalidation from publish webhooks, listed only translated URLs with hreflang alternates, and set lastmod from per-locale publish times. The shrink guard caught one bad generation in the first month. Search console showed the indexed share of submitted URLs rising from 58 to 87 percent over the following quarter.

Which Content Types Belong

Not every routable content type deserves a place in the sitemap. Include types whose pages carry unique, useful content: articles, products, documentation, landing pages, and category pages with real introductory text. Exclude types that exist mainly for navigation or internal purposes: tag archives with a handful of entries, author pages without biographies, search results, filtered and sorted listing variants, paginated pages beyond the first where the pagination adds nothing, and utility pages such as login or cart. Record the decision per content type in configuration, next to the robots and canonical rules for that type, so all three always agree. When a new content type is added to the CMS model, the pull request should include its sitemap decision; a CI check can fail if a routable type has none.

Editors sometimes need to exclude individual pages, such as campaign pages that should be reachable but not indexed. Give the SEO object a noindex flag, as described in modeling SEO fields, and make the sitemap query respect it. One flag controlling both the robots tag and sitemap inclusion keeps them consistent automatically.

The team also found that per-chunk regeneration made publishing faster to reflect: a new article appeared in its locale’s sitemap within seconds of publishing, instead of after the next nightly build, and crawler logs showed new articles being fetched noticeably sooner than before.

Frequently Asked Questions

Related

• Generating XML Sitemaps from Headless CMS Routes
• Incremental Sitemap Regeneration for Dynamic CMS Routes
• Handling Canonical URLs in Headless Multilingual Setups
• Robots.txt Configuration for Multi-Locale Headless Sites
• Automated SEO Audits for Headless CMS Deployments
• Handling 404s and Redirects in Headless Routing