Advertise next and previous pages with Link headers
Send the next, previous, first and last page URLs in a Link header, so clients follow links you build rather than assembling query strings themselves.
How-to › Section 3
Reading collections page by page, streaming, bulk jobs, file transfer, loading API data into notebooks, and scheduled pipelines that keep a copy in sync.
25 guides in 5 topics. Every guide compares its approaches and shows the output its commands printed.
Cursor versus offset versus keyset versus Link header (RFC 8288) pagination, on both sides: designing a list endpoint that survives inserts and deletes, page-size negotiation, stable ordering, total counts, backwards paging; and on the client, auto-pagination iterators, parallel page fetching with backpressure, paging over GraphQL connections.
Send the next, previous, first and last page URLs in a Link header, so clients follow links you build rather than assembling query strings themselves.
Compare the four ways to page a list on what decides it: whether a caller can miss a row when the data shifts, and what the query costs at page 900.
Put enough in the OpenAPI document that a generated client can walk a collection by itself, instead of handing every consumer a single-page call.
Encode the page position as base64url JSON and sign it with HMAC, so callers carry a cursor without reading it and a tampered one never reaches your query.
Wrap a cursor-paged list call in an async generator, so callers loop with for await, stop early without a stray request, and never handle a cursor.
Walk a paginated list endpoint to the end, read the three stop signals APIs actually send, and refuse to loop forever when a server repeats its cursor.
Move a list endpoint from offset to cursor pagination without breaking the clients still sending an offset, and show why the change is worth making.
Clamp an over-large page size to your ceiling rather than rejecting it, say in the response what was served, and refuse only values that are not sizes.
Wrap a paged list in an Enumerator that fetches pages as rows are pulled, so first(50) stays cheap, lazy.select stops early, and a 503 surfaces where you iterate.
Wrap a cursor-paged endpoint in a coroutine.wrap iterator so callers write one for loop, and raise a failed page as a table that `pcall` can catch.
Walk a paginated collection with a generator, so a caller writes one foreach, pages are fetched only as they are consumed, and an early break costs nothing.
Turn a cursor-paged endpoint into a sync and an async Python generator, take the first N items without fetching every page, and raise an error naming the page.
Server-Sent Events, WebSockets, chunked and NDJSON responses, long polling, resumable streams with last-event ids, backpressure, reconnection, and streaming LLM tokens to a client, from both the producer and consumer side.
Keep a slow sink from being overrun by a fast WebSocket feed: pause the socket in Node, read a WebSocketStream where it exists, or window acknowledgements.
Read an SSE endpoint that needs a bearer token or a POST body, which EventSource cannot send, and own reconnection, the retry field and Last-Event-ID yourself.
Give a streaming endpoint an iterator in TypeScript, Python, and Go that yields as lines arrive, closes the connection on cancel, and raises on a cut body.
Serve one job-progress feed as SSE, WebSocket, long poll, and short poll, restart the proxy in front of it, and choose by what survives and what each event costs.
Read an NDJSON or Server-Sent Events response line by line in sync and async Python, with a timeout per phase and a line splitter chosen on purpose.
Bulk import and export, batch endpoints, async jobs with status polling, multipart and resumable uploads (tus, S3 multipart), presigned URLs, and large downloads.
Answer a batch write with one result per input, keyed by the client's id, under an errors flag, so a client retries only what did not land.
Normalize an exported CSV, run it through a Bulk API 2.0 ingest job, then download the failed and unprocessed rows and prove every row has exactly one result.
Poll a job until it reaches a terminal state: honor Retry-After, back off with jitter when the server sends none, and stop at a deadline with the job id in hand.
Resume an upload from the last byte the server confirmed after a dropped connection or a closed tab, with tus, S3 multipart, Google resumable uploads, or your own.
For analysts: paginated endpoints into pandas and Polars DataFrames, DuckDB httpfs and JSON functions over API responses, caching responses on disk while exploring, rate limits inside a notebook, flattening nested JSON into columns (pandas.json_normalize, jq, struct), Observable and Deno notebooks for JavaScript.
Write a data loader that fetches a paginated API under Node at build time with a key from the environment, and read its JSON on the page with FileAttachment.
Generate a package where one call drains a paginated endpoint into a typed pandas frame, against a helper over `openapi-python-client` and a dlt source.
Flatten every page of Pydantic, `attrs`, or sdkgen objects into one DataFrame, with enums as plain strings and the row count checked against the objects read.
Incremental sync with cursor checkpoints, backfills under a rate limit, deduplication, change detection, quotas and cost, scheduling with cron, Airflow, Dagster and Prefect, extract tools (dlt, Airbyte, Singer taps, Meltano), landing in Parquet, DuckDB or a warehouse, and monitoring a sync.
Load a year of history in daily windows, checkpoint each so a rerun resumes, and pace calls to the published limit with a token bucket or the RateLimit field.
Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.
Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.