Content-Addressed Storage for Localized Media

This guide, part of Asset Duplication & CDN Sync, replaces name-based media storage with content addressing: every file is stored under the hash of its bytes, and a manifest maps each asset and locale to the hash it should use. Identical files are stored once no matter how many locales use them, URLs never change meaning, and every file can be cached forever.

Name-based storage is the default almost everywhere. An asset called hero.jpg is uploaded to /assets/hero.jpg, its German variant to /assets/de/hero.jpg, and a new version overwrites the old file at the same path. That model has three problems for localized sites. Many locale variants are byte-for-byte identical copies of the default, which wastes storage and transfer. Overwriting a path changes what a URL means, so every cache that holds it must be purged, and any cache that misses the purge serves the old file. And there is no cheap way to tell whether two variants are actually different.

The Problem

A software company with 18 locales stored a copy of every image under each locale’s folder, because the frontend built image URLs from the page’s locale. Of 31,000 stored files, fewer than 1,500 actually differed from the default. When the design team replaced a product screenshot, a script overwrote the file in all 18 folders and purged each path. Two locales’ purges failed silently, and those markets served the old screenshot for three days until the cache entries expired.

How Content Addressing Works

Objects are named by their hash. The storage key is the SHA-256 of the file’s bytes plus its extension, for example /objects/9f2e…c41a.jpg. Writing the same bytes twice produces the same key, so the second write is a no-op. Changing a single byte produces a new key.

Objects are immutable. An object’s URL always returns the same bytes, so it can be served with Cache-Control: public, max-age=31536000, immutable. Nothing ever needs to be purged at the object level.

A manifest gives meaning to hashes. The manifest maps (asset id, locale) to an object hash, along with metadata such as dimensions and alt text. Pages resolve image URLs through the manifest. Updating an asset means uploading a new object and changing the manifest entry; the only thing that changes is which hash a page references.

Pages are what you invalidate. Because objects never change, invalidation moves up one level: when a manifest entry changes, the pages that render that asset are revalidated, and they emit the new object URL.

Implementation

The upload function hashes the file, checks whether the object already exists and writes it only if not. The manifest is a small table in a database or key-value store, versioned with a revision number.

// lib/media/cas.ts
import { createHash } from "node:crypto";
import { S3Client, HeadObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({});
const BUCKET = process.env.MEDIA_BUCKET!;

export async function putObject(bytes: Buffer, ext: string, contentType: string): Promise<string> {
  const hash = createHash("sha256").update(bytes).digest("hex");
  const key = `objects/${hash}.${ext}`;
  try {
    await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));
    return key; // already stored: identical bytes, nothing to do
  } catch {
    await s3.send(new PutObjectCommand({
      Bucket: BUCKET,
      Key: key,
      Body: bytes,
      ContentType: contentType,
      CacheControl: "public, max-age=31536000, immutable",
    }));
    return key;
  }
}

// lib/media/manifest.ts
import { db } from "@/lib/db";

export interface ManifestEntry { assetId: string; locale: string; objectKey: string; width: number; height: number; alt: string; revision: number }

export async function setVariant(entry: Omit<ManifestEntry, "revision">): Promise<boolean> {
  const current = await db.manifest.findUnique({ where: { assetId_locale: { assetId: entry.assetId, locale: entry.locale } } });
  if (current?.objectKey === entry.objectKey && current.alt === entry.alt) return false; // nothing changed
  await db.manifest.upsert({
    where: { assetId_locale: { assetId: entry.assetId, locale: entry.locale } },
    create: { ...entry, revision: 1 },
    update: { ...entry, revision: (current?.revision ?? 0) + 1 },
  });
  return true;
}

export async function resolveVariant(assetId: string, chain: string[]): Promise<ManifestEntry | null> {
  for (const locale of chain) {
    const hit = await db.manifest.findUnique({ where: { assetId_locale: { assetId, locale } } });
    if (hit) return hit;
  }
  return null;
}

When setVariant returns true, the sync worker revalidates the pages tagged with asset:{assetId}. When a locale has no entry, resolveVariant walks the fallback chain, so a locale without its own variant uses the default object at no storage cost.

Garbage collection

Objects are never overwritten, so old ones accumulate. A weekly job lists objects, subtracts every key referenced by the manifest, including a few recent revisions kept for rollback, and deletes the rest after a grace period of several weeks. The grace period protects cached HTML that still references older objects. Because deletion is based on references, it is safe even when many locales shared an object: it is deleted only when nothing references it any more.

Configuration Reference

Gotchas & Edge Cases

• Hashing after transformation. Hash the exact bytes you store. If a pipeline converts or compresses images, hash the output, or two identical sources compressed differently will produce different keys and vice versa.
• Metadata is not in the hash. Alt text and captions belong in the manifest, not in the object. Changing alt text must not create a new object.
• Very large files. Hashing large videos takes time and memory. Stream them through the hash function and upload with multipart uploads.
• Direct links from outside. Hashed URLs change with every version, so external sites linking to an image will keep the old version. Offer stable alias URLs for assets intended to be linked externally, redirecting to the current object.

Worked Example

The software company migrated its 31,000 files by hashing each one and building the manifest from the existing folder structure. The object store ended up with 1,460 objects. Pages were changed to resolve images through the manifest with locale fallback chains. The next screenshot replacement produced one new object and 18 manifest updates, only three of which pointed at a different object, and revalidated 42 pages. There was nothing to purge at the CDN, so there was nothing to fail silently.

Choosing Between Content Addressing and Versioned Names

Content addressing is not the only way to get immutable URLs. Many CMSs already version asset URLs by including an upload id or revision in the path, which also makes caching safe. The difference is deduplication and control. Versioned names from the CMS change whenever anything is uploaded, even identical bytes, and each locale’s upload is a separate file; content addressing stores identical bytes once and lets you compare variants by hash. If your CMS’s asset CDN already serves versioned URLs and storage cost is not a concern, use it as is and skip this layer. If you run your own origins for residency, processing or cost reasons, content addressing is the cleanest model for them, and the manifest becomes a useful record of which locales actually have their own imagery, something editors and localization managers often ask for and rarely get.

Rollout Checklist

• Hash the stored bytes and write objects under their hash, skipping existing ones.
• Serve objects as immutable with year-long cache lifetimes.
• Build a manifest mapping asset id and locale to object key and metadata.
• Resolve images through the manifest with locale fallback chains.
• Revalidate pages, not objects, when manifest entries change.
• Garbage-collect unreferenced objects after a grace period.

Frequently Asked Questions

Related

• Asset Duplication & CDN Sync
• Syncing Localized Media Assets Across Global CDNs
• Purging Localized Assets by Cache Tag