Asset Duplication & CDN Sync

A headless CMS stores each media file once, but localized sites need locale-specific variants on regional edges — and copying assets naively produces cache fragmentation, runaway egress bills, and broken fallback chains. This guide covers the sync layer that keeps duplicated assets consistent: checksum-gated replication, deterministic path mapping, and surgical edge purges.

Integration Contract

The contract for localized assets has four parts. Identity: every asset has a stable id from the CMS, and every locale variant is identified by that id plus a locale code, never by a file name editors can change. Paths: a deterministic function maps id, locale and format to a storage path and a public URL, shared by the sync job, the frontend and the CDN configuration. Fallback: a declared chain per locale, such as fr-CA → fr → default, decides which variant is served when one is missing. Freshness: a publish of an asset or a variant triggers the sync and a purge of exactly the affected cache tags, with a scheduled reconciliation job as a backstop.

# .env: asset sync
ASSET_ORIGIN_EU=https://assets-eu.example.com
ASSET_ORIGIN_US=https://assets-us.example.com
ASSET_SYNC_CONCURRENCY=8
ASSET_FALLBACK_CHAIN="fr-CA:fr:default,de-AT:de:default,en-GB:en:default"
CDN_PURGE_TOKEN=scoped_purge_only_token

Why Cross-Locale Distribution Is Hard

The tension is decoupling asset storage from asset delivery. Editors upload once; the infrastructure must replicate, transform, and push locale-aware variants to regional origins without propagation lag or cache stampedes. That demands a sync layer that compares cryptographic checksums before transferring, normalizes locale path suffixes, and invalidates only the edges whose payloads actually changed. Get any of those wrong and you either serve stale media in secondary markets or pay to re-upload bytes that never moved. This is one piece of broader Localization & SEO Optimization, where media routing has to stay aligned with locale negotiation and hreflang.

Idempotent Duplication Pipelines

Trigger the pipeline from CMS webhooks or a scheduled reconciliation job. It must be idempotent: running it twice under identical conditions produces the same result — no duplicate files, no redundant transfers, no clobbering a newer variant. The implementation below fetches the source asset, computes a SHA-256 checksum, and uploads each locale variant with If-None-Match so the origin skips unchanged payloads.

import { createHash } from 'crypto';
import { fetch } from 'undici';

interface CMSAsset {
  id: string;
  url: string;
  locale: string;
  etag: string;
  metadata: Record<string, string>;
}

interface SyncConfig {
  targetOrigin: string;
  maxRetries: number;
  concurrencyLimit: number;
}

async function computeChecksum(buffer: Buffer): Promise<string> {
  return createHash('sha256').update(buffer).digest('hex');
}

async function fetchWithRetry(url: string, retries: number = 3): Promise<Response> {
  for (let attempt = 0; attempt < retries; attempt++) {
    try {
      const res = await fetch(url);
      if (res.ok) return res;
      if (res.status >= 500) {
        await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
        continue;
      }
      throw new Error(`Fetch failed for ${url}: ${res.status}`);
    } catch (err) {
      if (attempt === retries - 1) throw err;
      await new Promise(r => setTimeout(r, Math.pow(2, attempt) * 1000));
    }
  }
  throw new Error('Max retries exceeded');
}

async function syncAssetToEdge(asset: CMSAsset, config: SyncConfig): Promise<void> {
  const response = await fetchWithRetry(asset.url, config.maxRetries);
  const buffer = await response.arrayBuffer();
  const checksum = await computeChecksum(Buffer.from(buffer));
  
  const contentType = response.headers.get('content-type') ?? 'application/octet-stream';
  const extension = asset.url.split('?')[0].split('.').pop();
  const localeSuffix = asset.locale !== 'default' ? `.${asset.locale}` : '';
  const targetPath = `/assets/${asset.id}${localeSuffix}.${extension}`;
  const targetUrl = `${config.targetOrigin}${targetPath}`;

  // Conditional PUT: the origin stores the checksum as the object's ETag and answers
  // 412 Precondition Failed when an object with that ETag already exists.
  const uploadRes = await fetch(targetUrl, {
    method: 'PUT',
    headers: {
      'Content-Type': contentType,
      'X-Asset-Checksum': checksum,
      'If-None-Match': `"${checksum}"`,
    },
    body: buffer,
  });

  if (uploadRes.status === 412) {
    console.log(`[SKIP] Asset ${asset.id} already up-to-date at ${targetPath}`);
    return;
  }

  if (!uploadRes.ok) {
    throw new Error(`Upload failed for ${targetPath}: ${uploadRes.status} ${uploadRes.statusText}`);
  }

  console.log(`[SYNC] Successfully deployed ${asset.id} to ${targetPath}`);
}

The pipeline below shows how the checksum gate keeps the sync idempotent — unchanged payloads never cross the wire.

The If-None-Match header plus server-side ETag validation is what makes this cheap: a 412 short-circuits the write, so unchanged payloads are never stored twice. Checking the target’s ETag with a HEAD request first also avoids sending the bytes at all. At scale, Syncing localized media assets across global CDNs extends this with a versioned manifest and webhook-driven queueing.

Cache Coordination & Fallback

CDN providers expose cache tags, surrogate keys, and soft-purge so you can invalidate a single asset variant instead of flushing a directory. While a localized asset is still propagating, the routing layer should serve the default variant — not a 404, which breaks responsive image pipelines and triggers a cache-miss cascade.

When an edge node gets a request for a locale variant that hasn’t arrived, it should serve the cached default or proxy to the primary origin and fetch the variant in the background. That prevents a stampede during a high-traffic launch. Pairing stale-while-revalidate with max-age lets the edge serve a slightly stale asset during the sync window while a background fetch pulls the new checksum — the mechanics are in HTTP Conditional Requests. Fallback behavior across routes is covered in Content Fallback & Routing.

Serving the right variant at the edge

The page usually knows its locale and can emit the variant’s URL directly, which is the simplest and most cacheable option. When it cannot, for example for assets referenced from rich text that was written once for all locales, an edge function can rewrite a neutral asset URL to the best available variant using the fallback chain and a manifest of which variants exist.

// edge/asset-variant.ts: rewrite /media/{id}.{ext}?locale=fr-CA to the best existing variant
import manifest from "./variant-manifest.json"; // { "hero-42": ["default", "fr", "de"] }

const CHAINS: Record<string, string[]> = { "fr-CA": ["fr-CA", "fr", "default"], "de-AT": ["de-AT", "de", "default"] };

export default {
  async fetch(req: Request): Promise<Response> {
    const url = new URL(req.url);
    const match = url.pathname.match(/^\/media\/([\w-]+)\.(\w+)$/);
    if (!match) return fetch(req);
    const [, id, ext] = match;
    const locale = url.searchParams.get("locale") ?? "default";
    const available = (manifest as Record<string, string[]>)[id] ?? ["default"];
    const chosen = (CHAINS[locale] ?? [locale, "default"]).find((l) => available.includes(l)) ?? "default";
    const suffix = chosen === "default" ? "" : `.${chosen}`;
    return fetch(new URL(`/assets/${id}${suffix}.${ext}`, url.origin), req);
  },
};

The query parameter keeps the cache key explicit: each locale has its own cached response, and a missing variant resolves to its fallback instead of a 404.

Path Normalization

Duplication pipelines break at the seam between storage paths and URL routing. If the CMS emits locale-prefixed URLs (/fr/assets/logo.webp) but the CDN stores flat paths (/assets/logo.fr.webp), image references 404 and SEO signals degrade. Normalize paths deterministically during sync so every variant maps to a known route.

Generate a build-time asset manifest that ties CMS asset IDs to locale codes, CDN paths, and fallback hierarchy, then feed it to both the frontend and the CDN config so route resolution and delivery can’t drift. Route Mapping for Multilingual Sites covers keeping URL structure, asset paths, and locale negotiation aligned. Apply locale suffixes consistently and sanitize path-traversal characters to prevent cache poisoning; a CI lint step that resolves every duplicated asset to a valid route catches drift before deploy.

Deciding Where Variants Live

There are three places a localized variant can live, and the choice affects cost, latency and complexity. In the CMS’s own asset store and CDN, as a localized asset field: the simplest option, with no sync layer at all, and adequate for most sites whose traffic is served well by the CMS’s global CDN. On your own regional origins, synced from the CMS: needed for data residency, custom processing, strict cost control or when assets must sit behind your own domain and security rules. Generated on the fly by an image service from a master asset and locale parameters, for example overlaying translated text on a template: attractive for large numbers of locales with small differences, but it moves complexity into rendering and makes caching depend on parameter hygiene.

Many sites combine them. Product photos stay in the CMS’s CDN because they are identical everywhere. Legal PDFs are synced to regional origins because regulators expect them to be hosted in-region. Campaign banners with translated headlines are generated from a template by an image service, cached at the edge by locale. Decide per asset kind, record the decision in the asset model, and let the sync layer act only on the kinds that need it.

Cost Model

Every duplicated variant costs storage in each region, egress when it is copied, and CDN cache space that competes with other content. A site with 20,000 images, 12 locales and 3 regions can end up with over 700,000 stored objects if everything is duplicated, most of them byte-for-byte identical. Three measures keep this in check. Deduplicate by content: store objects under their checksum, so identical payloads are stored once per region regardless of how many locales reference them, as described in content-addressed storage for localized media. Duplicate only what differs: most images need no locale variant at all, and the fallback chain serves the default. Expire what is unused: variants for retired campaigns and removed locales should be deleted by lifecycle rules once no entry references them. Chart stored objects and egress per region monthly; growth that outpaces content growth means one of these measures is missing.

Regional replication also has a latency benefit that is worth measuring rather than assuming. For sites whose traffic is concentrated in one region, a single origin behind a global CDN with tiered caching often performs just as well as regional origins, at a fraction of the operational effort. Run the comparison with real traffic before building multi-region replication.

Preview & Draft Assets

Assets have drafts too. An editor who replaces the German variant of a campaign banner expects to see it in preview before it is published, and must not see it appear on the live site early. Keep draft asset variants out of the sync until the asset or the entry referencing it is published: the CMS’s asset URLs for drafts, served with the preview token, are good enough for preview. When the publish webhook arrives, sync the variant and purge its tag. Some CMSs publish assets independently of entries, which means an asset can go live before the page that uses it; that is usually harmless, but for embargoed material, restrict asset publishing to the same role that publishes the page.

Error Handling & Resilience

The sync layer sits between the CMS and several origins, so partial failure is the normal case: one region’s origin times out while the others succeed. Treat each target as an independent task with its own retries, and record per-target status in a manifest keyed by asset id and checksum. A reconciliation job compares the manifest with the CMS’s asset list every hour and retries anything missing or stale, so a failed webhook or a regional outage heals itself. Never delete the previous variant until the new one is confirmed on every target; serving an old banner for a few minutes is better than serving a broken image. Alert on reconciliation findings that persist across several runs, which indicate a systematic problem such as an expired origin credential rather than a transient failure.

Testing the Sync Layer

Test the path function in isolation with a table of ids, locales and formats, including tricky inputs such as locale codes with regions, ids with unusual characters and assets without an extension. Test the sync job against local fake origins that can return 412, 5xx and timeouts, and assert that each case leads to the right outcome: skip, success with purge, or retry. In staging, publish a test asset with variants in three locales and verify from each region that the right variant is served, that the fallback applies to a fourth locale, and that a republish purges only the changed variant. Keep this end-to-end check as a scheduled probe in production too.

Observability & Hardening

Emit structured logs per sync attempt — asset ID, checksum, target origin, HTTP status — and track sync latency, cache hit ratio, checksum-mismatch rate, and retry exhaustion. These signals surface propagation bottlenecks before users hit them.

Cost compounds fast: every locale variant multiplies storage and egress. Lifecycle policies that archive unused variants, content-addressable storage that deduplicates identical payloads across regions, and CDN tiered caching keep the bill down. Scope invalidations with Cache Tags and Purging so a publish purges one variant, not a region.

For hardening, put a rate-limited worker pool with exponential backoff between the webhook and the origin so a publish flood can’t saturate it, and canary each sync to one edge region first to validate checksums, routing, and cache behavior before rolling out globally. Asset duplication is a distributed-systems problem, not a file copy.

Worked Example

A travel company served 14 locales from a CMS whose asset CDN was fine for most images, but its legal documents had to be hosted in the EU and its campaign banners contained translated text. The first version of its sync copied every asset to three regional buckets on every publish. Transfer costs grew monthly, and after a large campaign the German banner showed the French headline for an hour, because the purge covered the default path but not the locale variant. The team rebuilt the layer along the lines of this topic: variants only for banners and legal documents, checksum-gated writes, content-addressed storage, locale-specific cache tags and hourly reconciliation. Transfer fell by over 95 percent, and the next campaign’s banners appeared correctly in every market within a minute of publishing.

The lesson the team took from the incident was that the purge, not the copy, is where localized asset pipelines usually fail. Every variant needs its own tag, and every publish needs to purge exactly those tags and nothing broader.

Frequently Asked Questions

Related

• Syncing Localized Media Assets Across Global CDNs
• Content-Addressed Storage for Localized Media
• Localizing Images with Embedded Text
• Purging Localized Assets by Cache Tag
• Localization & SEO Optimization