# How to stream a long HTTP response in Python with httpx

> Read an NDJSON or Server-Sent Events response line by line in sync and async Python, with a timeout per phase and a line splitter chosen on purpose.

Source: https://voxgig.com/howto/stream-a-response-with-python-httpx

- Audience: api-consumer
- Level: beginner
- Languages: python
- Verified: 2026-11-20
- Published: 2026-11-20

## Short answer

Open the response with httpx's client.stream(), read iter_bytes() or aiter_bytes(), and split on the newline byte yourself before parsing each line as JSON. Set httpx.Timeout per phase so a quiet stream outlives the five-second default. Use httpx-sse for text/event-stream, which exposes each event's id and the server's retry delay for a resume. A generated SDK's stream helper saves the parsing only when it reads the body as it arrives.

---
## You will need

Python 3.10 or later, or 3.11 for the OpenAPI Generator client, and an endpoint that answers NDJSON
or Server-Sent Events. The sample's
`npm ci` installs its pinned PyPI packages into `pydeps/`, so Node 22 runs that one step. The
sample's `api.json` offers `GET /readings` as a JSON array or as NDJSON, and `GET /readings/live`
as [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html).

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [httpx client.stream](https://www.python-httpx.org/quickstart/) | NDJSON or plain text, in sync or async code | You own the framing, the timeouts, and the error when a line is cut | The API ships a client that streams |
| [httpx-sse](https://github.com/florimondmanca/httpx-sse) | The endpoint answers `text/event-stream` | One more dependency, and reconnection is still your loop to write | The body is NDJSON |
| [Kiota](https://learn.microsoft.com/en-us/openapi/kiota/quickstarts/python) | You already call the API through a Kiota client | Its typed `get()` buffers the body, so streaming means leaving the request builder | You only need the stream, not the typed models |
| [OpenAPI Generator](https://openapi-generator.tech/docs/generators/python/) | You already ship its Python client | Its `*_without_preload_content()` method hands back an unread urllib3 response to parse | You want httpx and async |
| [requests `iter_lines`](https://requests.readthedocs.io/en/latest/api/) | Sync code that already uses requests | A 512-byte read size that holds short lines back unless the body is chunked | You need async, or lines on time |
| [sdkgen streaming feature](https://github.com/voxgig/sdkgen/blob/main/docs/reference/features.md) | You call the API through an sdkgen Python SDK | `stream('list')` iterates a list parsed after the whole body arrived | Readings must reach you while the response is open |

Voxgig maintains sdkgen. It is one of six options here, not the recommendation.

httpx by hand gives you control of every timeout and every byte, in exchange for writing the
framing yourself. The generated clients save you that code, and all three here wait for the
body before they parse it. requests sits between, with a read size you have to
know about.

## Check what the generated SDK yields

Generate the Python SDK with the streaming feature, as `generate.sh` does. Every entity has a
`stream()` method with or without it; the feature takes over the iteration once a client sets it
active:

```bash title="generate.sh"
npx --yes @voxgig/create-sdkgen@0.31.9 meter --def api.json -t py -f streaming
```

`demo.py` reads five readings, sent 50 ms apart, three ways. Each row records how many readings
the server had sent when that one reached the caller:

```bash
python3 demo.py
```

```text output
sdkgen Reading().stream('list')
  r1   1.5 kWh   server had sent 5 of 5
  r2  1.75 kWh   server had sent 5 of 5
  r3   2.0 kWh   server had sent 5 of 5
  r4   2.5 kWh   server had sent 5 of 5
  r5  3.25 kWh   server had sent 5 of 5
httpx client.stream() and iter_bytes(), sync, URL from sdk.prepare()
  r1   1.5 kWh   server had sent 1 of 5
  r2  1.75 kWh   server had sent 2 of 5
  r3   2.0 kWh   server had sent 3 of 5
  r4   2.5 kWh   server had sent 4 of 5
  r5  3.25 kWh   server had sent 5 of 5
httpx AsyncClient.stream() and aiter_bytes()
  r1   1.5 kWh   server had sent 1 of 5
  r2  1.75 kWh   server had sent 2 of 5
  r3   2.0 kWh   server had sent 3 of 5
  r4   2.5 kWh   server had sent 4 of 5
  r5  3.25 kWh   server had sent 5 of 5
```

The SDK's `stream('list')` is a generator over a list that its `requests` transport has already
parsed, so every reading arrives after the fifth was sent. It still earns its place for a short
list, because a caller writes the same loop as for a long one. When the readings must arrive on
time, take the URL from the SDK's `prepare()`, which builds the request and sends nothing, and
stream it with httpx.

## Read each line as it arrives

`client.stream()` returns the response before the body is read, and the `with` block closes the
connection when the loop ends or raises:

```python title="stream_lines.py"
# One timeout per phase. The read timeout is the longest gap allowed between two
# chunks, not the length of the whole response, so a long export that keeps sending
# chunks does not trip it.
TIMEOUT = httpx.Timeout(connect=5.0, read=60.0, write=5.0, pool=5.0)
NDJSON = {"accept": "application/x-ndjson"}


def split_ndjson(chunks):
    """Split on b"\\n" alone, as NDJSON specifies, and parse each line.

    iter_lines() splits like str.splitlines(), which also breaks on U+2028 and
    other separators that are legal inside a JSON string.
    """
    buffered = b""
    for chunk in chunks:
        buffered += chunk
        *lines, buffered = buffered.split(b"\n")
        for line in lines:
            if line.strip():
                yield json.loads(line)
    if buffered.strip():
        raise ValueError(f"the stream ended mid-line: {buffered[:20]!r}")


def iter_readings(client: httpx.Client, url: str, headers=None):
    with client.stream("GET", url, headers={**NDJSON, **(headers or {})}) as response:
        response.raise_for_status()
        yield from split_ndjson(response.iter_bytes())
```

The async reader is the same code with `httpx.AsyncClient`, `async with`, and
[`aiter_bytes()`](https://www.python-httpx.org/async/). Each phase of
[`httpx.Timeout`](https://www.python-httpx.org/advanced/timeouts/) has its own limit, and `read`
is the one a quiet stream meets.

## Resume an event stream from the last id

[httpx-sse](https://github.com/florimondmanca/httpx-sse) parses `text/event-stream` and leaves
reconnection to you. Its README says so, and gives each event's `id` and `retry` as the values to
reconnect with:

```python title="watch_sse.py"
def watch_readings(client: httpx.Client, url: str, state: dict):
    """Yield readings. `state` keeps the last event id and the server's retry delay."""
    headers = {}
    if state.get("last_id") is not None:
        headers["last-event-id"] = state["last_id"]
    with connect_sse(client, "GET", url, headers=headers) as source:
        for sse in source.iter_sse():
            if sse.retry is not None:
                state["retry_ms"] = sse.retry
            if sse.event != "reading":
                continue
            state["last_id"] = sse.id
            yield json.loads(sse.data)
```

```bash
python3 resume.py
```

```text output
  r1  first connection
  r2  first connection
reconnect with last-event-id r2; the server asked for a retry delay of 2000 ms
  r3  second connection
  r4  second connection
  r5  second connection
last-event-id the server saw: [None, 'r2']
```

## Send Kiota's request with stream=True

Kiota's Python `readings.get()` calls `send_collection_async` and returns parsed `Reading`
objects, and its `/readings/live` builder returns the whole body as `bytes`. Kiota's
[abstractions page](https://learn.microsoft.com/en-us/openapi/kiota/abstractions) describes a
response handler that can return the native HTTP response instead. `kiota_stream.py` takes a
shorter route. It builds the request with the generated `to_get_request_information()`, turns it
into an `httpx.Request` with the adapter's `convert_to_native_async()`, and sends it with
`stream=True`:

```bash
python3 kiota_stream.py
```

```text output
readings.get(): 5 Reading objects, server had sent 5 of 5
readings.live.get(): bytes
the same request as an httpx.Request, accept: application/x-ndjson
  server had sent, as each reading arrived: [1, 2, 3, 4, 5]
```

`convert_to_native_async()` runs the adapter's authentication provider before it builds the
request, so the hand-off keeps Kiota's URL template and credentials. Kiota is the better choice
when the rest of your code already talks to the API through its fluent builders.

## Leave the OpenAPI Generator response unread

OpenAPI Generator's Python client generates `list_readings_without_preload_content()`, which
returns the urllib3 response unread. Unless you pass your own `Accept`, the serializer offers both
media types to `select_header_accept()`:

```python title="other/openapi_client/api/default_api.py"
        # set the HTTP header `Accept`
        if not any(key.lower() == 'accept' for key in _header_params):
            _header_params['Accept'] = self.api_client.select_header_accept(
                [
                    'application/json', 
                    'application/x-ndjson'
                ]
            )
```

That function returns the first entry containing `json`, so the request asks for the array:

```python title="other/openapi_client/api_client.py"
        for accept in accepts:
            if re.search('json', accept, re.IGNORECASE):
                return accept

        return accepts[0]
```

`oag_stream.py` passes `_headers={"Accept": "application/x-ndjson"}` and reads the response with
urllib3's `stream(1024)`:

```bash
python3 oag_stream.py
```

```text output
list_readings(): 5 Reading objects, server had sent 5 of 5, accept: application/json
list_readings_without_preload_content(), accept: application/x-ndjson
  server had sent, as each reading arrived: [1, 2, 3, 4, 5]
```

The generated package asks for Python 3.11 and Pydantic. It suits a team that ships that
generator's clients in several languages and wants no second HTTP library.

## Check it worked

`check.py` records `time.monotonic()` as each reading arrives and compares it with the server's
count:

```bash
python3 check.py
```

```text output
ok  sdkgen stream('list') yields all five within 50 ms, after the last was sent
ok  httpx sync: one reading per send, first to last at least 0.5 s apart
ok  httpx async: one reading per send, first to last at least 0.5 s apart
ok  a read timeout shorter than a quiet gap raises ReadTimeout
ok  the per-phase TIMEOUT reads the same quiet stream to the end
ok  a body that ends mid-line raises
ok  split_ndjson keeps a line that carries U+2028 in a string
ok  httpx iter_lines() splits that line in two
ok  httpx-sse resumes from Last-Event-ID with no repeats
ok  requests iter_lines() hands over each line of a chunked body as it lands
ok  requests iter_lines() holds back a short body that ends at the close
ok  requests iter_lines() with chunk_size=1 hands over that body's lines as they land
ok  Kiota readings.get() returns after the last reading was sent
ok  Kiota's request sent by httpx with stream=True yields each reading as it is sent
ok  OpenAPI Generator list_readings() asks for JSON and returns after the last reading
ok  OpenAPI Generator's unread response, read in chunks, yields each reading as it is sent
```

The server sends a reading every 150 ms in the first three checks, so a reader that buffers fails
the spread. Each failed check prints `FAIL` and makes the script exit 1.

## When it goes wrong

```bash
python3 pitfall.py
```

```text output
httpx default timeout: Timeout(timeout=5.0)
quiet for 0.6 s, read timeout 0.3 s: ReadTimeout after ['r1', 'r2']
requests ITER_CHUNK_SIZE: 512
requests iter_lines(), chunked body, server's count per line:    [1, 2, 3, 4, 5]
requests iter_lines(), body ends at close, server's count per line: [5, 5, 5, 5, 5]
iter_lines() on a reading with U+2028 in a string: ['r1', 'JSONDecodeError', 'JSONDecodeError', 'r3', 'r4', 'r5']
split_ndjson(iter_bytes()) on the same body:       ['r1', 'r2', 'r3', 'r4', 'r5']
```

The stream stops with `ReadTimeout` during a pause. The httpx default allows five seconds between
chunks. Set `read` to more than the longest pause the API documents, and keep `connect` short.

`requests` shows nothing, then every line at once. `iter_lines()` reads 512 bytes at a time. A
chunked body hands over each chunk as it lands, but a body that ends at the connection's close
returns a read only once 512 bytes arrive or the connection closes. Pass `chunk_size=1`, which
reads a byte at a time, or use httpx.

A valid line fails to parse. httpx's `iter_lines()` splits like
[`str.splitlines()`](https://docs.python.org/3/builtins/stdtypes.html#str.splitlines), which breaks on
U+2028, a character JSON allows inside a string without an escape.[^1] Split on `b"\n"` yourself.

## When not to do this

Do not stream a response that is a few kilobytes and arrives in one round trip. `response.json()`
is shorter, and the sdkgen `stream('list')` loop costs nothing at that size.

Do not hand-write an SSE parser when httpx-sse covers the format. The parser is the easy part to
get almost right, and the README carries a reconnection example built on `id` and `retry`.

Do not set `timeout=None` to make a quiet stream survive. It removes the connect and pool limits
too, so a dead host hangs the caller. Raise `read` alone.

Pick Kiota or OpenAPI Generator when the rest of the integration already uses their typed
models, and stream through their request builders for the one endpoint that needs it.

## Related how-tos

- [Retry a streaming response without replaying received data](/howto/retry-a-streaming-response-without-replaying-data)

- [Consume Server-Sent Events with fetch and custom headers](/howto/consume-sse-with-fetch-and-custom-headers)

- [Pick SSE, WebSockets, or long polling for a live endpoint](/howto/choose-sse-websockets-or-long-polling)

- [Offer sync and async clients from one Python SDK](/howto/sync-and-async-clients-from-one-python-sdk)

- [Expose a streaming response through a generated SDK](/howto/expose-a-streaming-response-through-a-generated-sdk)

## Last verified

Verified 2026-11-20 against Python 3.13.18, Node 22.23.5, httpx 0.28.1, httpx-sse 0.4.3,
requests 2.34.2, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, Kiota 1.38.0 with its Python
packages 1.18.0, and OpenAPI Generator 7.27.0 with Pydantic 2.14.4. Every output block is what the
command preceding it printed. The generators ran once, through `generate.sh`, and their output is
committed and run.

[^1]: [RFC 8259](https://www.rfc-editor.org/rfc/rfc8259.html) lists the characters a JSON string
must escape, and U+2028 and U+2029 are not among them. JavaScript string literals refused both
until the [JSON superset proposal](https://tc39.es/proposal-json-superset/) allowed them, which
TC39's [list of finished proposals](https://github.com/tc39/proposals/blob/main/finished-proposals.md)
places in ECMAScript 2019. Python's `str.splitlines()` counts both as line boundaries.