Paginating Large Hygraph Collections for Static Builds

This guide belongs to Hygraph GraphQL Content Federation and covers fetching large collections from Hygraph during static builds and for sitemaps: thousands of products, articles or locations. It explains why a single large query fails, how connection queries with cursors paginate reliably, why stable ordering matters, how to keep list queries lean, how to bound concurrency and handle rate limits, and how to avoid fetching everything on every build.

Hygraph limits how many entries a single request returns and how complex a query may be. A build that asks for all products with their relations in one query hits those limits as the catalogue grows, first with truncated results, which are easy to miss, then with errors. Pagination solves it, but naive pagination has traps of its own: offset-based pages shift when entries are published during the build, parallel requests trigger rate limiting, and fetching full entries for listing pages multiplies the work.

The Problem

A property portal generated a static page per listing, about 14,000 of them, with a build script that requested listings(first: 20000) with every field and relation. For a while it worked; then builds started finishing with far fewer pages than expected, because the API returned fewer entries than requested without any error the script checked for. After the team switched to offset pagination with skip, builds occasionally contained duplicate pages and missed others, because agents published new listings during the build, shifting every later page by the number of new entries.

How Pagination Works in Hygraph

Connection queries. Every model has a connection query, such as listingsConnection, returning edges with nodes, pageInfo with hasNextPage and endCursor, and an aggregate with the total count. Cursor-based pagination with first and after continues exactly where the previous page ended.

Stable ordering. Cursors are only meaningful with a stable order. Order by a unique, immutable field such as id, or by a timestamp with the id as a tie-breaker. Ordering by a field that editors change, such as a title, lets entries move between pages during the build.

Lean list queries. The list query collects ids, slugs and the fields listing pages show. Detail pages fetch full entries separately, with their own queries.

Bounded concurrency. Detail fetches run in parallel with a small limit, and requests that receive a 429 response wait and retry with backoff.

Counting and checking. The aggregate.count tells the build how many entries to expect; compare it with what was collected and fail the build on a mismatch.

Implementation

A paginated collector for listing summaries, with rate-limit handling:

// build/listings.ts
const LISTINGS_PAGE = /* GraphQL */ `
  query ListingsPage($first: Int!, $after: String, $stage: Stage!) {
    listingsConnection(first: $first, after: $after, orderBy: id_ASC, stage: $stage) {
      pageInfo { hasNextPage endCursor }
      aggregate { count }
      edges { node { id slug title city price updatedAt } }
    }
  }
`;

type Summary = { id: string; slug: string; title: string; city: string; price: number; updatedAt: string };

async function gql<T>(query: string, variables: Record<string, unknown>, attempt = 0): Promise<T> {
  const res = await fetch(process.env.HYGRAPH_ENDPOINT!, {
    method: "POST",
    headers: { "Content-Type": "application/json", Authorization: `Bearer ${process.env.HYGRAPH_READ_TOKEN}` },
    body: JSON.stringify({ query, variables }),
  });
  if (res.status === 429 && attempt < 6) {
    const wait = Number(res.headers.get("retry-after")) * 1000 || 500 * 2 ** attempt;
    await new Promise((r) => setTimeout(r, wait));
    return gql<T>(query, variables, attempt + 1);
  }
  const json = await res.json();
  if (!res.ok || json.errors) throw new Error(`Hygraph: ${json.errors?.[0]?.message ?? res.status}`);
  return json.data as T;
}

export async function allListingSummaries(): Promise<Summary[]> {
  const out: Summary[] = [];
  let after: string | null = null;
  let expected = 0;
  do {
    const data: any = await gql(LISTINGS_PAGE, { first: 100, after, stage: "PUBLISHED" });
    const conn = data.listingsConnection;
    expected = conn.aggregate.count;
    out.push(...conn.edges.map((e: { node: Summary }) => e.node));
    after = conn.pageInfo.hasNextPage ? conn.pageInfo.endCursor : null;
  } while (after);

  // Entries published during the build can change the count slightly; large gaps mean a bug.
  if (Math.abs(out.length - expected) > Math.max(10, expected * 0.01)) {
    throw new Error(`Collected ${out.length} listings, expected about ${expected}`);
  }
  return out;
}

Detail fetches run with a concurrency limit, so the build uses the API steadily instead of in bursts:

// build/pool.ts
export async function mapPool<T, R>(items: T[], limit: number, fn: (item: T) => Promise<R>): Promise<R[]> {
  const results = new Array<R>(items.length);
  let next = 0;
  async function worker() {
    while (next < items.length) {
      const i = next++;
      results[i] = await fn(items[i]);
    }
  }
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
  return results;
}

// usage: const details = await mapPool(summaries, 6, (s) => getListing(s.slug));

In Next.js, generateStaticParams can return the slugs from the summaries, and each page fetches its own details. Pages beyond the most visited can render on demand instead of at build time.

Page size

Choose a page size that the API accepts and that keeps responses small, such as 100 entries with a lean selection. Check your project’s limits; larger pages save requests but increase the risk of hitting complexity limits when fields are added to the list query later.

Incremental builds

Fetching everything on every build wastes time once the collection is large. Store the latest updatedAt seen in the previous build and query only entries updated since then, with a where: { updatedAt_gt: $since } filter, to decide which pages to regenerate. Combine it with on-demand revalidation from webhooks, and rebuild everything only on a schedule or after schema changes, so an incremental mistake cannot persist indefinitely.

Configuration Reference

Gotchas & Edge Cases

• Silent truncation. Requests for more entries than allowed may return fewer without an error; always paginate and compare counts.
• Localized collections. With locales, entries without content in the requested locales are skipped; count per locale.
• Draft builds. Preview deployments that build from DRAFT see more entries than production; do not compare their counts with published counts.
• Remote fields in lists. Remote fields in list queries call the remote API per entry; keep them out of list selections.
• Build timeouts. Long builds on hosted platforms may time out; move rarely visited pages to on-demand rendering.

Worked Example

The property portal switched to connection queries ordered by id, a lean summary selection, detail fetches with a concurrency of six, 429 handling and a count check. The next build that would have been truncated failed loudly instead, which revealed a filter bug. Builds became correct and faster, and the team then moved listings older than a year to on-demand rendering. Build duration for 14,000 listings fell from 38 minutes with the offset approach to 11 minutes, with no duplicates or gaps.

Sitemaps and Feeds for Large Collections

Sitemaps are the other place where large collections are fetched in full. Generate them from the same summary collector, which already returns slugs and updatedAt, rather than from separate queries. Split sitemaps into files of a manageable size with a sitemap index, and cache them with a short lifetime, revalidated when webhooks report publishes or unpublishes. Feeds usually need only the most recent entries, so query them with orderBy: publishedAt_DESC and a small first instead of paginating the whole collection. Keeping all list-style outputs on the same collector means one tested, rate-limit-aware code path for every place that needs many entries at once.

Rollout Checklist

• Replace large single queries with connection queries and cursors.
• Order by a unique, immutable field.
• Keep list selections lean and fetch details separately.
• Bound concurrency and handle 429 responses with backoff.
• Compare collected counts with the aggregate count.
• Use incremental and on-demand rendering for very large collections.

Frequently Asked Questions

Related

• Hygraph GraphQL Content Federation
• Federating Remote Data with Hygraph Remote Sources
• Cursor Pagination Merge Functions for CMS Lists in Apollo