Monitoring CMS Webhook Delivery, Failures and Retries

This guide completes Webhook-Triggered Rebuilds with observability: the signals that show whether publishes are reaching the live site, the alerts that catch silent failures before editors do, and a single number, publish-to-live latency, that summarizes the health of the whole pipeline.

Webhook pipelines fail quietly. A rotated secret makes every delivery return 401; the CMS retries a few times and gives up, and nothing on the site changes until an editor notices that yesterday’s correction is still missing. A build hook starts returning 429 during a campaign; the handler logs it and moves on. Because each component “works” from its own point of view, only end-to-end monitoring reveals the gap.

The Problem

A publisher rotated its webhook secret as part of a routine security change. The CMS’s webhook configuration still held the old secret, so every delivery failed signature verification. The handler correctly returned 401, the CMS retried three times per event, and then marked the deliveries as failed in a log nobody watched. For four days, publishes appeared only when the hourly time-based revalidation happened to run on a visited page; quiet pages stayed stale for the whole period. The failure was discovered when a reader pointed out an outdated statement on a policy page.

Nothing was broken in the usual sense: the handler did exactly what it should with invalid signatures. What was missing was anyone, or anything, watching the rate of rejections.

How to Monitor the Pipeline

Four kinds of signals, collected at four points:

Delivery. Most CMSs keep a webhook delivery log with the status code, response time and retry count of each attempt, and several expose it through the management API. Pull it periodically, or at least check it during incidents. A rising share of non-2xx responses is the earliest sign of trouble.

Decisions. The handler should log one structured line per event with its outcome: verified or rejected (and why), filtered as draft, deduplicated, queued. These lines turn “the site did not update” into “the event was rejected for a bad signature at 09:14”.

Reactions. The worker logs each revalidation or build trigger with its result, and the queue exposes its depth and oldest message age. A growing queue or repeated failures from the build platform point at the reaction layer.

Outcome. A synthetic probe publishes a small change to a dedicated test entry on a schedule, then polls the public URL until the change appears, and records the elapsed time. That is the publish-to-live latency, and it is the one metric that proves the pipeline works end to end, regardless of what each component reports.

Implementation

The handler emits one structured log line per event; the probe runs as a scheduled job. Both are small.

// lib/webhook-log.ts: one line per event, in a shape log tools can aggregate
type Outcome = "rejected_signature" | "rejected_replay" | "filtered_draft" | "duplicate" | "queued";

export function logWebhook(fields: { outcome: Outcome; cms: string; env: string; entryId?: string; event?: string; ms: number }): void {
  console.info(JSON.stringify({ kind: "cms_webhook", ts: new Date().toISOString(), ...fields }));
}

// jobs/publish-probe.ts: runs every 15 minutes
import contentful from "contentful-management";

const PROBE_URL = "https://www.example.com/status/publish-probe";
const TIMEOUT_MS = 120_000;

export async function runProbe(report: (metric: { latencyMs: number | null; ok: boolean }) => Promise<void>): Promise<void> {
  const client = contentful.createClient({ accessToken: process.env.CMA_TOKEN ?? "" });
  const env = await (await client.getSpace(process.env.CONTENTFUL_SPACE ?? "")).getEnvironment("master");
  const stamp = new Date().toISOString();

  const entry = await env.getEntry(process.env.PROBE_ENTRY_ID ?? "");
  entry.fields.stamp = { "en-US": stamp };
  const started = Date.now();
  await (await entry.update()).publish();

  while (Date.now() - started < TIMEOUT_MS) {
    const html = await (await fetch(PROBE_URL, { headers: { "Cache-Control": "no-cache" } })).text();
    if (html.includes(stamp)) {
      await report({ latencyMs: Date.now() - started, ok: true });
      return;
    }
    await new Promise((r) => setTimeout(r, 5000));
  }
  await report({ latencyMs: null, ok: false });
}

The probe page is an ordinary page rendered through the same pipeline as content pages, with the same caching, revalidation tags and CDN in front, which is what makes the measurement meaningful. Exclude it from sitemaps and search indexing, and keep the test entry in a content type that editors do not see in their daily work.

Building the dashboard

One dashboard with four panels covers the pipeline. The first shows webhook outcomes per hour, stacked by outcome, so a sudden band of rejected_signature stands out immediately. The second shows queue depth and oldest message age. The third shows action results, revalidations and builds, with failures highlighted. The fourth shows the probe’s publish-to-live latency as a line with its alert threshold. Put the most recent CMS delivery failures in a table below, pulled from the delivery log where the CMS exposes it. Share the latency panel with the editorial team as well: it answers “is publishing working?” for them without anyone having to ask engineering.

Configuration Reference

Gotchas & Edge Cases

• Alerting on single failures. Individual deliveries fail for transient reasons all the time, and the CMS retries them. Alert on rates and consecutive failures, or the alert will be muted within a week.
• Probe entries triggering real builds. On static sites, each probe publish triggers a build. Run the probe less often, or map the probe content type to a lightweight revalidation path.
• Logging payloads. Webhook bodies can contain draft content and personal data. Log ids and outcomes, not bodies.
• Monitoring only production. Staging pipelines break too, and a broken staging pipeline hides the next production problem during testing. Run a cheaper probe on staging as well.
• Ignoring the CMS side. Your logs cannot show deliveries that never reached you. The CMS’s delivery log is the only place where “the event left the CMS but never arrived” is visible.

Worked Example

After the four-day incident, the publisher added the structured handler log, an alert on signature rejections and the publish probe every 15 minutes. Three months later, a CDN configuration change accidentally cached the probe page for an hour. The probe alert fired 30 minutes after the change, well before any editor noticed, and the fix was deployed within the hour. The latency dashboard became a regular part of the weekly editorial meeting, where “live in about eight seconds” replaced “give it a few minutes” as the answer to when changes appear.

Rollout Checklist

• Add a structured log line for every webhook outcome in the handler.
• Export queue depth, oldest message age and action results from the worker.
• Create a probe entry, probe page and scheduled probe job, and chart its latency.
• Configure alerts on rates and consecutive failures, not single events.
• Review the CMS delivery log after every secret rotation or endpoint change.

Frequently Asked Questions

Related

• Webhook-Triggered Rebuilds
• Routing CMS Webhooks to Staging, Production and Preview Environments
• Preview & Draft Workflow Patterns