# How to paginate an API with Python generators

> Turn a cursor-paged endpoint into a sync and an async Python generator, take the first N items without fetching every page, and raise an error naming the page.

Source: https://voxgig.com/howto/paginate-an-api-with-python-generators

- Audience: api-consumer
- Level: intermediate
- Languages: python
- Verified: 2026-11-21
- Published: 2026-11-21

## Short answer

Write a Python generator that requests one page, yields its items, and follows the `next` cursor until it is null, so callers iterate with a plain for loop. Write the same walk as an async generator over `httpx.AsyncClient`. Take the first N with `itertools.islice`, which fetches only the pages it needs, and raise an exception that carries the page number and cursor. Give each consumer its own generator, because one is used up after a single pass.

---
## You will need

Python 3.11 or later, requests, httpx, and an API whose list endpoint returns a cursor. Verified
2026-11-21 against Python 3.13.18, requests 2.34.2, httpx 0.28.1, `@voxgig/sdkgen` 4.44.3 and
`openapi-python-client` 0.30.0. The document, `api.json`, has one operation, `GET /books`, taking
`cursor` and `limit` and answering `{ "items": [...], "next": "c3" }`, with `next` null on the
last page. `server.py` stands in for it on 127.0.0.1.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [boto3 get_paginator](https://docs.aws.amazon.com/boto3/latest/guide/paginators.html) | You call AWS, which ships a paginator per operation | A Paginator and PageIterator shape that covers AWS operations and nothing else | The API is not an AWS service |
| [Hand-written generator with requests or httpx](https://docs.python.org/3/reference/expressions.html#yieldexpr) | Any cursor API, any client, and callers who should see one `for` loop | You own the loop guard, the error type and the async twin | A generated SDK already pages correctly for your API |
| [`openapi-python-client`](https://github.com/openapi-generators/openapi-python-client) | You want typed models and will write the loop around them | No paging code at all, so the generator on this page is still yours | You want paging generated with the client |
| [sdkgen paging feature](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md) | You generate several languages and want one paging signal in each | An opt-in feature whose control object you must pass and patch by hand | Your body names its cursor `next`, unless you add the wrapper here |
| [Speakeasy x-speakeasy-pagination](https://www.speakeasy.com/docs/sdks/customize/runtime/pagination) | You generate with Speakeasy and can add an extension to the document | An account to generate, and a `while` loop over `next()` instead of a `for` | You cannot add vendor extensions to the published document |

Voxgig maintains sdkgen. It is one of five options here, and the one with a gap this page works
around.

A hand-written generator costs the most code and works against any client. boto3's paginator is
reusable where a generator is not, but it covers AWS alone. Stainless is not a row: on May 18,
2026 it announced it is [winding down all hosted Stainless products](https://www.stainless.com/blog/stainless-is-joining-anthropic/),
its SDK generator included, with no new signups.

## Yield books from each page

Two generators split the job. `pages()` owns the HTTP and the cursor; `books()` flattens pages
into items, so a caller never sees a page boundary.[^1]

```python title="pages.py"
def pages(session, base, limit=3):
    """Yield one page of books at a time, with requests. Nothing is fetched until the caller asks."""
    cursor, page, used = None, 1, set()
    while True:
        params = {"limit": limit} if cursor is None else {"limit": limit, "cursor": cursor}
        r = session.get(f"{base}/books", params=params, timeout=10)
        if r.status_code != 200:
            raise PageError(page, cursor, r.status_code, r.text)
        body = r.json()
        yield body["items"]
        cursor = _next_cursor(page, cursor, body, used)
        if cursor is None:
            return
        used.add(cursor)
        page += 1


def books(session, base, limit=3):
    """Yield one book at a time, across every page."""
    for items in pages(session, base, limit):
        yield from items
```

The raise sits inside the generator, so it surfaces at the caller's `for` line, on the item after
the last good page. `PageError` carries the page number and the cursor that was sent for it.
`_next_cursor` raises too when the server repeats a cursor, which would otherwise loop forever.

`apages()` is the same body with `async def` and `await client.get(...)` over an
[`httpx.AsyncClient`](https://www.python-httpx.org/async/), and `abooks()` flattens it with
`async for`.

## Stop early and see which page failed

`demo.py` runs four walks against the stand-in and prints every query it received.

```bash
python3 demo.py
```

```text output
for loop over books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: 3, limit=3 | limit=3&cursor=c3 | limit=3&cursor=c6
islice(books(), 4)
  books: b1 b2 b3 b4
  requests: 2, limit=3 | limit=3&cursor=c3
async for over abooks(), break after 5
  books: b1 b2 b3 b4 b5
  requests: 2, limit=3 | limit=3&cursor=c3
page 2 answers 503
  books before the error: b1 b2 b3
  raised: page 2 (cursor 'c3') failed: 503 {"code": "unavailable", "message": "try again"}
  err.page=2 err.cursor='c3' err.status=503
```

[`islice`](https://docs.python.org/3/library/itertools.html#itertools.islice) stopped after four
books, so the third page was never requested. The async walk has no `islice`, so it breaks out of
the loop inside [`contextlib.aclosing`](https://docs.python.org/3/library/contextlib.html#contextlib.aclosing),
which closes the async generator at once instead of leaving it to the garbage collector. The
failed walk delivered page 1 before raising, and the exception names page 2 and `c3`, which is
the request to retry.

## Follow the cursor with sdkgen's paging feature

The sdkgen Python target scaffolded with `-f paging` reads the page signal and records it on the
control dict, the second argument of `list()`. The feature sends a recorded `cursor` back as the
cursor parameter, and records a body field named `next` as `next`, which it does not send. Its
options `cursorParam`, `pageParam` and `limitParam` name query parameters only, so no setting
points the feature at a different response field. `sdks.py` shows both, then the wrapper that copies one into the other.

```bash
python3 sdks.py
```

```text output
openapi-python-client list_books.sync()
  returns BookPage: 3 items, next_='c3'
  requests: limit=3
openapi-python-client, wrapped in opc_books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: limit=3 | cursor=c3&limit=3 | cursor=c6&limit=3
sdkgen paging feature, passing the same ctrl three times
  call 1: b1 b2 b3
  call 2: b1 b2 b3
  call 3: b1 b2 b3
  ctrl paging after: next='c3' cursor=None hasMore=True
  requests: limit=3&page=1 | limit=3&page=1 | limit=3&page=1
sdkgen, wrapped in sdkgen_books()
  books: b1 b2 b3 b4 b5 b6 b7
  requests: limit=3&page=1 | cursor=c3&limit=3 | cursor=c6&limit=3
```

With this document, the loop the [features reference](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md)
shows, `while ctrl.paging.hasMore`, never ends: `hasMore` stays true and each call asks for
`page=1`. `@voxgig/sdkgen` 4.45.0, the newest release, generated the same Python tree and
repeated the same page. A body that names its cursor `nextCursor`, `cursor` or `next_cursor`
fills `cursor` instead, and `check.py` runs the documented loop to the last page on each.

```python title="generated_pages.py"
    ctrl, page, used = {"paging": {}}, 1, set()
    while True:
        sent = ctrl["paging"].get("cursor")  # a failed call rewrites ctrl["paging"], so keep it first
        try:
            items = sdk.Book().list({"limit": limit}, ctrl)
        except Exception as err:
            raise PageError(page, sent, getattr(err, "status", None), err) from err
        for entity in items:
            yield entity.data_get()
        nxt = ctrl["paging"].get("next")
        if nxt is None:
            return
        if nxt in used:
            raise PageError(page + 1, nxt, "loop", "the server sent a cursor it already sent")
        used.add(nxt)
        ctrl["paging"]["cursor"] = nxt
        page += 1
```

The feature also adds `page=1` to the first request, which this stand-in ignores as an undeclared
parameter. A server that refuses unknown parameters answers 400 instead.

## Page with Speakeasy's extension or a wrapped Python client

`openapi-python-client` generates `list_books.sync()` and `sync_detailed()`, which return one
`BookPage` per call. Following `next` is left to you. It renames the field `next_`, as the first block of
`sdks.py` shows.[^2] `opc_books()` wraps `sync_detailed()` in the same generator shape and raises
`PageError` on any response that did not parse as a `BookPage`, including a 503 the document does
not list.

Speakeasy was not run, because generating needs an account. Its
[pagination page](https://www.speakeasy.com/docs/sdks/customize/runtime/pagination) describes
`x-speakeasy-pagination` with `type: cursor`, an input with `name`, `in: parameters` and
`type: cursor`, and an output `nextCursor` written as a JSONPath, here `$.next`. The generated
response gains `next()`, and the page's loop, `while response is not None: response =
response.next()`, stops when `next()` returns nothing. Wrap that loop in a generator to give
callers a `for`.

## Check it worked

`check.py` asserts every claim on this page and exits 1 on any failure.

```bash
python3 check.py
```

```text output
ok   books() walks every page in order
ok   a generator fetches only the pages its consumer reads
ok   the async generator walks and stops the same way
ok   a failed page raises PageError naming page 2 and its cursor
ok   the stand-in refuses what api.json does not allow
ok   a second consumer of one generator gets nothing
ok   a repeated cursor raises instead of looping
ok   openapi-python-client, wrapped, walks every page
ok   sdkgen paging needs next copied to cursor, and loses it on a failure
ok   sdkgen's documented loop advances on nextCursor, cursor and next_cursor
0 failed
```

## When it goes wrong

The second consumer of a generator gets nothing. A generator runs once, so a count followed by a
list comprehension over the same object reads it dry on the first pass.

```bash
python3 pitfall.py
```

```text output
one generator, two consumers
  count: 7
  titles: []
  requests: 3
a fresh generator per consumer
  count: 7
  titles: 7, first 'Emma', last 'Kindred'
  requests: 6
```

Call `books()` once per consumer, as the second block does, at the price of a second walk. Use
[`itertools.tee`](https://docs.python.org/3/library/itertools.html#itertools.tee) only when both
consumers advance together, since it buffers whatever one reads ahead of the other.

A retried sdkgen walk starts over. After a failed call, `ctrl["paging"]` holds `cursor=None` and
`hasMore=False`, so a loop that catches the error and tests `hasMore` stops early. Keep the cursor
before each call, as `sdkgen_books()` does.

The walk never ends. A server that repeats a cursor loops a naive generator forever, so keep the
cursors already followed and raise on a repeat.

## When not to do this

Do not hand-write the loop when the client you use already pages correctly. A boto3 paginator, or
a Speakeasy SDK whose document carries `x-speakeasy-pagination`, is less code to keep. If you
already hold a Speakeasy account and can add the extension, take Speakeasy: its generated `next()`
follows the cursor with no wrapper.

Do not turn on sdkgen's paging feature and trust its loop without running it against your own
response shape. This page needed a wrapper because the cursor field is named `next`.

Choose `openapi-python-client` when typed models matter more than generated paging; the generator
here is fifteen lines on top of it. Do not return a generator from a function whose callers expect
a list they can read twice.

## Related how-tos

- [Loop over every page of a REST collection](/howto/loop-over-every-page-of-a-collection)

- [Paginate an API with PHP generators](/howto/paginate-an-api-with-php-generators)

- [Expose paged results as an async iterator in TypeScript](/howto/paged-results-as-an-async-iterator-in-typescript)

- [Describe pagination so generators can follow it](/howto/describe-pagination-in-openapi-for-generators)

- [Stream a long HTTP response in Python with httpx](/howto/stream-a-response-with-python-httpx)

## Last verified

Verified 2026-11-21 against Python 3.13.18, requests 2.34.2, httpx 0.28.1, httpx2 2.16.2,
`@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3, `@voxgig/apidef` 8.32.4 and
`openapi-python-client` 0.30.0. Every output block is what the command preceding it printed,
against a stand-in on 127.0.0.1, and Python 3.12.3 printed the same four blocks. The two generated
clients were generated once and committed, and `@voxgig/sdkgen` 4.45.0 was run once on the same
document to repeat the paging test.
Speakeasy was not run, and `x-speakeasy-pagination` is described from its pagination page.

[^1]: Generators arrived in Python 2.2 through [PEP 255](https://peps.python.org/pep-0255/),
created in May 2001 by three authors. The asynchronous kind
needed [PEP 525](https://peps.python.org/pep-0525/), created in July 2016 for Python 3.6, so
`async for` over a generator is fifteen years younger than `for`. boto3 describes its own
[Paginator](https://docs.aws.amazon.com/boto3/latest/guide/paginators.html) as reusable, the one
property neither PEP gives a generator.

[^2]: The generator appends an underscore to every name in Python's `builtins` module, which its
[`strings.py`](https://github.com/openapi-generators/openapi-python-client/blob/main/openapi_python_client/strings.py)
builds from `dir(builtins)`. The [built-in functions](https://docs.python.org/3/builtins/functions.html)
include `next()`, the call that advances an iterator, so the cursor becomes `next_`. The list
exempts one builtin, `id`, which is why `Book.id` keeps its name.