Incremental Sitemap Regeneration for Dynamic CMS Routes

Regenerating a full sitemap on every deploy stops scaling past a few thousand dynamic routes — it exhausts CMS API quotas, triggers cascading CDN invalidations, and blocks the pipeline. Incremental regeneration isolates route-level updates: a versioned route registry, webhook-triggered delta fetches, and chunked XML where only modified chunks get rewritten and purged. It extends Dynamic Sitemap Generation for sites where full regeneration no longer fits in a build.

The architecture decouples route enumeration from XML serialization. Instead of iterating every CMS document per deploy, maintain a persistent versioned registry; webhook payloads trigger targeted fetches, and only modified slugs, locales, or content types enter the regeneration queue.

Three Failure Modes at Scale

Sitemap generation in headless environments fails three ways:

1. API rate-limit exhaustion: Bulk enumeration during build hits CMS API ceilings, causing timeouts and failed deploys.
2. Edge cache staleness: Aggressive max-age on sitemap-index.xml makes the CDN serve a stale index, delaying crawler discovery.
3. Phantom locale routes: Fallback configs emit 200 OK URLs with duplicate or placeholder content, diluting crawl budget.

All three converge on the same fix: event-driven incremental updates, not monolithic rebuilds.

Deterministic Route Resolution

Start with a resolver that queries the CMS delivery API using cursor pagination for consistent traversal, filtering strictly on updatedAt to capture only deltas. This TypeScript utility shows the incremental fetch with typing, error boundaries, and cursor management:

import { CMSClient, RouteEntry } from './types';

interface FetchOptions {
  cmsClient: CMSClient;
  lastSyncTimestamp: string;
  batchSize?: number;
}

export async function fetchIncrementalRoutes({
  cmsClient,
  lastSyncTimestamp,
  batchSize = 100,
}: FetchOptions): Promise<RouteEntry[]> {
  const routes: RouteEntry[] = [];
  let cursor: string | null = null;

  try {
    do {
      const response = await cmsClient.getEntries({
        limit: batchSize,
        cursor,
        fields: ['slug', 'locale', 'updatedAt', 'contentType'],
        filter: `updatedAt > "${lastSyncTimestamp}"`,
      });

      const mappedRoutes = response.items.map((item) => ({
        path: `/${item.locale}/${item.slug}`,
        lastmod: item.updatedAt,
        priority: item.contentType === 'landing' ? 0.8 : 0.6,
        changefreq: 'weekly' as const,
        entryId: item.sys.id,
        locale: item.locale,
      }));

      routes.push(...mappedRoutes);
      cursor = response.nextCursor ?? null;
    } while (cursor);
  } catch (error) {
    console.error('Route enumeration failed:', error);
    throw new Error('Incremental fetch interrupted. Verify CMS API health.');
  }

  return routes;
}

The output feeds a chunked sitemap generator. The Sitemaps Protocol caps each file at 50,000 URLs and 50MB uncompressed, so chunk the registry by locale and content type. Give each chunk its own ETag for conditional requests. The index keeps referencing every chunk; only the lastmod of modified chunks changes, which tells crawlers which files to fetch again.

Deletions and Unpublishes

A delta fetch filtered by updatedAt returns entries that changed and are still published. It does not return entries that were deleted or unpublished, because they are no longer in the delivery API at all. Those removals must come from webhook events: on entry.unpublish and entry.delete, remove the entry’s routes from the registry and rewrite the chunks that contained them. Keep a periodic full reconciliation, weekly for example, that compares the registry with a complete enumeration and fixes anything a missed webhook left behind. Without it, removed pages linger in the sitemap and crawlers keep requesting URLs that return 404 or 410.

Stable Chunk Assignment

Incremental regeneration only saves work if a changed route always lands in the same chunk. Assign routes to chunks with a stable function, such as locale plus content type plus a hash of the entry id modulo the number of chunks, instead of filling chunks sequentially by date or alphabet. Sequential filling moves every later route to a different chunk when one is inserted, which rewrites all of them. With hashing, a new article changes one chunk. Choose the number of chunks per locale and type so each stays well below the size limits as content grows, and change it rarely, since changing it reshuffles everything once.

Webhook Deduplication

CMS platforms emit duplicate entry.update events — draft transitions, scheduled publishing, and metadata-only edits all fire redundant payloads. Derive an idempotency key from entryId plus a SHA-256 hash of the modified field paths, and track processed keys in an LRU cache over a 30–60s window. A matching key short-circuits the cycle, so only substantive mutations trigger serialization.

Locale Fallback

Missing localized variants must be omitted from the sitemap or mapped to a canonical parent — never indexed as low-value fallback pages. Route Mapping for Multilingual Sites keeps hreflang and canonical tags synced with the XML. When generating locale chunks, cross-reference available translations against the fallback hierarchy; exclude missing variants from the sitemap but keep them in the routing layer for graceful degradation. This underpins Dynamic Sitemap Generation at global scale.

CDN Synchronization

Issue targeted PURGE requests keyed on ETag or Last-Modified rather than purging directories. Cache tags that map to sitemap chunks let you invalidate surgically without touching static assets. Set Cache-Control: public, max-age=3600, stale-while-revalidate=86400 on updated chunks to balance freshness against edge performance. Conditional requests — see MDN on HTTP ETag — mean crawlers download only modified XML, cutting bandwidth and speeding indexation.

Pipeline Flow

The full sequence routes each webhook through deduplication before any serialization work happens, so only substantive mutations rewrite a chunk.

The full sequence:

1. Event Ingestion: CMS webhooks deliver payload metadata to a serverless function or edge worker.
2. Deduplication Check: Idempotency keys are validated against an LRU cache. Redundant events are dropped.
3. Delta Fetch: The route resolver queries the CMS API using updatedAt filters and cursor pagination.
4. Chunk Serialization: Modified routes are grouped by locale/content type, serialized to XML, and assigned ETag values.
5. Index Reconciliation: The sitemap-index.xml keeps every chunk and updates the lastmod of modified ones.
6. Edge Invalidation: Targeted cache tags or URL paths are purged. New chunks deploy with optimized Cache-Control directives.

Incremental regeneration turns sitemap management from a deployment bottleneck into a background, event-driven process. Isolating route updates, enforcing deduplication, and using edge cache primitives is what holds crawl efficiency steady as content velocity climbs.

Observability for Incremental Sitemaps

An incremental system can drift silently, so give it a few metrics. Record, for every webhook, whether it was dropped as a duplicate, which chunks it rewrote and how long the rewrite took. Chart the number of routes per chunk to spot chunks approaching the size limit. Most importantly, record the reconciliation results: the number of routes added, removed or changed by the weekly full comparison. In a healthy system that number is small and stable; a sudden rise means webhooks are being lost or mishandled, and the incremental path can no longer be trusted until the cause is fixed.

Worked Example

A classifieds site with 80,000 listings in two locales regenerated its sitemap nightly, which took 40 minutes, consumed a large share of its API quota and still left new listings out of the sitemap for up to a day. Moving to a route registry, delta fetches on publish webhooks, removal on unpublish and hashed chunk assignment reduced a typical update to two API requests and one rewritten chunk, visible within seconds. A weekly reconciliation found a handful of discrepancies each week from missed webhooks and fixed them automatically.

Rollout Checklist

• Keep a persistent route registry keyed by entry id and locale.
• Fetch deltas on publish webhooks, filtered by update time.
• Remove routes on unpublish and delete webhooks.
• Assign routes to chunks with a stable hash.
• Keep every chunk in the index and update lastmod of changed ones.
• Reconcile the registry with a full enumeration weekly.

Frequently Asked Questions

Related

• Dynamic Sitemap Generation
• Generating XML Sitemaps from Headless CMS Routes
• Webhook-Triggered Rebuilds