How-to › Move data in and out

How to turn typed SDK response objects into a pandas DataFrame#

Flatten every page of Pydantic, `attrs`, or sdkgen objects into one DataFrame, with enums as plain strings and the row count checked against the objects read.

Audience
API consumer
Level
beginner
Topic
Load API data into notebooks and DataFrames
Languages
Python
Verified

pd.DataFrame(page.data) on a page of OpenAPI Generator models returns seven columns named 0 to 6, and every cell is a pair such as ('id', 'ord_101'). On sdkgen entity objects it returns one column of objects. Dump the models first and the status column prints OrderStatus.PAID in every group-by. A loop that stops after the first page hands you three orders out of seven.

What you get

You will end up with one DataFrame built from every page of a list endpoint, with plain strings in its enum column and a row count checked against the objects read. This is for you if a vendor’s Python SDK hands you model objects rather than dictionaries.

Short answer

To turn typed SDK responses into a pandas DataFrame, read every page, convert each object to a dict, and flatten the list with pd.json_normalize. On OpenAPI Generator’s Pydantic models, call model_dump() and map the enum members with .value. On openapi-python-client models, call to_dict(), which leaves UNSET fields out. sdkgen’s py-data package returns the frame from one call. Then check that df.shape[0] equals the objects read.

You will need

Python 3.12 or later, pandas, and a Python SDK for the API installed from PyPI. Verified 2026-11-20 against Python 3.13.18, pandas 3.1.1, Pydantic 2.14.4, OpenAPI Generator 7.27.0, openapi-python-client 0.30.0, and @voxgig/sdkgen 4.44.3. The document, api.json, describes an order list behind a cursor envelope of data, next_cursor, and has_more, seven orders in three pages. Every script runs against mock_api.py, a local stand-in that refuses a cursor it did not issue.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
OpenAPI Generator python models with model_dump()The SDK is OpenAPI Generator’s python client, with Pydantic modelsA paging loop you write, Enum members to map, and float columns where integers go missingAnalysts want every page in one call with nullable integer columns
openapi-python-client models with to_dict()The SDK was generated by openapi-python-client and returns attrs modelsA paging loop you write, dates as strings to parse, and UNSET wherever to_dict() is bypassedYou want Pydantic validation on every record
sdkgen py-data methodThe API owner generates with sdkgen and publishes the py-data package beside the SDKpandas only, a second package to install, and string columns for dates until you parse themThe vendor publishes no py-data package, or the analysts work in Polars
Speakeasy Python models with model_dump()The vendor’s SDK is a Speakeasy Python SDK whose list operation has pagination configuredThe same Enum and datetime handling as any Pydantic model, plus a next() loop per endpointThe vendor’s document has no pagination extension, so next() has nothing to follow

Voxgig maintains sdkgen. It is one of four options here, not the recommendation.

The two Pydantic rows keep the SDK’s validation until the last step, and a dict per object is the price. The to_dict() that openapi-python-client generates writes strings for enums and dates, so its frame needs one to_datetime and no enum mapping. The py-data package skips the object layer entirely, but it exists only where the API owner generated and published it.

See what the DataFrame constructor makes of objects

The constructor treats a list of objects as a list of rows, and what a row turns into depends on the class.

PYTHONPATH=pydeps:generated/py:generated/py-data:generated/oag:generated/opc python3 one_column.py
sdkgen OrderEntity objects: shape (3, 1), columns [0], dtype object
OpenAPI Generator pydantic models: shape (3, 7), columns [0, 1, 2, 3, 4, 5, 6]
  first cell: ('id', 'ord_101')

The sdkgen entity is opaque to pandas, so each row is one object. A Pydantic model is iterable, and iterating it yields (field_name, field_value) pairs, so pandas builds a column per field and stores the pairs.1 Group-bys and joins need a column per field, so each object has to become a dict first.

Read every page through the sdkgen SDK

generate.sh built all three clients once, from the same api.json, and the output is committed under generated/. The sdkgen part is two commands: the scaffold adds the py target, the py-data target that wraps it, and the paging feature.

(cd "$WORK" && npx --yes @voxgig/create-sdkgen@0.31.9 orderdesk --def api.json \
  --target py --target py-data --feature paging)
(cd "$WORK/orderdesk-sdk/.sdk" && npm run generate)

The py SDK’s list() returns OrderEntity objects, and data_get() unwraps each one into a dict. The paging feature writes the server’s signals into the control dict you pass, so passing the same dict again fetches the next page.

def sdkgen_records(sdk) -> list[dict]:
    """Every order through the sdkgen py SDK, unwrapped with data_get()."""
    orders = sdk.Order()
    ctrl = {"paging": {}}
    records: list[dict] = []
    seen: set[str] = set()
    while True:
        records.extend(order.data_get() for order in orders.list(None, ctrl))
        if not ctrl["paging"].get("hasMore"):
            return records
        cursor = ctrl["paging"].get("cursor")
        if cursor is not None and cursor in seen:
            raise RuntimeError(f"pagination repeated cursor {cursor!r}")
        seen.add(cursor)

sdkgen_frame.py runs that loop, then the py-data method, which drains the same pages and returns the frame itself.

PYTHONPATH=pydeps:generated/py:generated/py-data:generated/oag:generated/opc python3 sdkgen_frame.py
py SDK, data_get() per entity: 7 objects, 7 rows, 3 requests
py-data, ad.orders(): 7 rows, 3 requests
  GET /v1/orders?page=1
  GET /v1/orders?cursor=cur_0003
  GET /v1/orders?cursor=cur_0006

        id    status  total  items             placed_at customer.country
0  ord_101      paid   42.5      3  2026-05-04T09:15:00Z               NG
1  ord_102   shipped   18.0      1  2026-05-04T08:40:00Z               NL
2  ord_103   pending   7.25   <NA>  2026-05-03T17:05:00Z               TW
3  ord_104  refunded   99.0      4  2026-05-03T11:30:00Z               IL
4  ord_105      paid  12.75      2  2026-05-02T20:00:00Z               KE
5  ord_106   shipped   64.0      5  2026-05-02T10:10:00Z               NG
6  ord_107      paid   30.0      2  2026-05-01T07:55:00Z               IN

id                   string
items                 Int64
note                 string
placed_at            string
status               string
total               Float64
customer.country        str
customer.name           str

placed_at with parse_dates: datetime64[us, UTC]

The first request carries page=1, because the feature sends a page number until a response hands it a cursor; the stand-in ignores the parameter. The dtypes come from the entity model, which records status as a $STRING and keeps none of its four enum values, so this frame has no Enum members to map. items stays an integer column with <NA> for the order that left it out, and placed_at stays text until parse_dates names it.

Dump Pydantic models from OpenAPI Generator and Speakeasy

OpenAPI Generator’s python client has no paging helper; a search of the generated package for paginat finds nothing. The loop is yours, and model_dump() turns each model into a dict.

def frame_from_pydantic(models: list) -> pd.DataFrame:
    """model_dump() keeps datetimes and Enum members; turn the members into strings."""
    df = pd.json_normalize([m.model_dump() for m in models])
    df["status"] = df["status"].map(lambda member: member.value)
    return check_rows(df, len(models))
PYTHONPATH=pydeps:generated/py:generated/py-data:generated/oag:generated/opc python3 oag_frame.py
7 models from 3 requests, 7 rows

                         model_dump()        status mapped
id                                str                  str
status                            str                  str
total                         float64              float64
items                         float64              float64
placed_at         datetime64[us, UTC]  datetime64[us, UTC]
note                              str                  str
customer.name                     str                  str
customer.country                  str                  str

status[0] after model_dump(): <OrderStatus.PAID: 'paid'>
status[0] after .value:       'paid'
placed_at[0]:                 2026-05-04 09:15:00+00:00
model_dump(mode='json'): status str, placed_at str

The two dtypes columns are identical, which is the trap: the enum column reads str while it holds OrderStatus members.2 placed_at arrived as datetime objects, and pandas 3.1 typed them on its own, so no to_datetime call was needed here. model_dump(mode="json") gives plain strings for both, which trades the enum fix for a date column you parse yourself. json_normalize turned the nested customer into customer.name and customer.country.

Speakeasy’s Python SDKs use Pydantic for every generated model, so the same frame_from_pydantic applies. Its paging is the x-speakeasy-pagination extension in the vendor’s document, with type: cursor and an outputs.nextCursor path, and the response gains a next() method that returns nothing after the last page.3 Speakeasy was not run for this page, since generation needs an account.

Check it worked

test_frames.py runs every route against the stand-in and compares each frame’s row count with the objects iterated and the stand-in’s request count.

    def test_the_pydantic_route_reads_every_page_and_maps_the_enum(self):
        models = oag_models(self.oag())
        df = frame_from_pydantic(models)
        self.assertEqual((len(models), df.shape[0], len(self.api.seen)), (ORDER_COUNT, ORDER_COUNT, PAGES))
        self.assertTrue(all(type(s) is str for s in df["status"]))
        self.assertEqual(str(df["placed_at"].dtype), "datetime64[us, UTC]")
PYTHONPATH=pydeps:generated/py:generated/py-data:generated/oag:generated/opc python3 test_frames.py
test_a_cursor_that_repeats_stops_the_loop (__main__.Guards.test_a_cursor_that_repeats_stops_the_loop) ... ok
test_a_cursor_that_repeats_stops_the_sdkgen_loop (__main__.Guards.test_a_cursor_that_repeats_stops_the_sdkgen_loop) ... ok
test_check_rows_refuses_a_short_frame (__main__.Guards.test_check_rows_refuses_a_short_frame) ... ok
test_the_entity_model_keeps_status_as_a_plain_string (__main__.Guards.test_the_entity_model_keeps_status_as_a_plain_string) ... ok
test_attrs_asdict_leaks_unset_into_the_frame (__main__.Routes.test_attrs_asdict_leaks_unset_into_the_frame) ... ok
test_model_dump_leaves_enum_members_in_a_str_column (__main__.Routes.test_model_dump_leaves_enum_members_in_a_str_column) ... ok
test_py_data_returns_every_page_as_one_frame (__main__.Routes.test_py_data_returns_every_page_as_one_frame) ... ok
test_py_data_types_columns_from_the_model (__main__.Routes.test_py_data_types_columns_from_the_model) ... ok
test_the_attrs_route_reads_every_page_without_sentinels (__main__.Routes.test_the_attrs_route_reads_every_page_without_sentinels) ... ok
test_the_constructor_makes_one_object_column (__main__.Routes.test_the_constructor_makes_one_object_column) ... ok
test_the_pydantic_route_reads_every_page_and_maps_the_enum (__main__.Routes.test_the_pydantic_route_reads_every_page_and_maps_the_enum) ... ok
test_the_sdkgen_py_sdk_loop_reads_every_page (__main__.Routes.test_the_sdkgen_py_sdk_loop_reads_every_page) ... ok
test_the_stand_in_refuses_cursors_and_limits_it_did_not_document (__main__.Routes.test_the_stand_in_refuses_cursors_and_limits_it_did_not_document) ... ok
test_without_paging_py_data_stops_after_one_page (__main__.Routes.test_without_paging_py_data_stops_after_one_page) ... ok

Fourteen tests print ok and the runner ends with OK. check_rows raises ValueError when df.shape[0] differs from the count, so a frame built from a short loop fails where it is made.

When it goes wrong

The group-by labels read OrderStatus.PAID in place of paid. model_dump() keeps the Enum member, and printing one shows its class. Map the column with .value, as frame_from_pydantic does.

PYTHONPATH=pydeps:generated/py:generated/py-data:generated/oag:generated/opc python3 pitfall.py
1. an enum column holds Enum members, and its dtype says str
   dtype: str, type of a value: OrderStatus
   status
   OrderStatus.PAID        3
   OrderStatus.PENDING     1
   OrderStatus.REFUNDED    1
   OrderStatus.SHIPPED     2
   after .map(lambda member: member.value):
   status
   paid        3
   pending     1
   refunded    1
   shipped     2

2. without the paging feature, the first page is the whole frame
   rows: 3 of 7, requests: ['/v1/orders']

3. attrs.asdict() keeps the UNSET sentinel that to_dict() leaves out
   attrs.asdict(): items object, row 3 items = Unset
   to_dict():      items float64, row 3 items = nan

The frame holds three rows of seven. The client was constructed without feature={"paging": {"active": True}}, so py-data made one request and treated it as the only page. Pass the feature, and keep the row-count check.

An items column turns to object and arithmetic on it raises. attrs.asdict() copied the Unset sentinel that openapi-python-client puts in absent fields. Call the model’s own to_dict(), which leaves them out, so pandas fills NaN.

When not to do this

Do not convert objects to a frame inside a service that only filters and forwards the records. The SDK’s types are the point there, and a frame throws them away for a table you never group.

Do not expect sdkgen’s py-data package from a vendor that ships only an OpenAPI Generator or Speakeasy SDK. It exists when the API owner generates with sdkgen and publishes it; otherwise the model_dump() loop here is the route.

Do not choose the py-data package for analysts who work in Polars. pandas is the only frame library it knows. A Speakeasy SDK with pagination configured is the better source there, because next() and model_dump() feed any frame library.

Do not trust dtypes to show an enum problem. The column reads str while it holds Enum members, and only a value check, as in the test, finds them.

Last verified

Verified 2026-11-20 against Node 22.23.5, Python 3.13.18, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, @voxgig/apidef 8.32.4, OpenAPI Generator 7.27.0, openapi-python-client 0.30.0, pandas 3.1.1, Pydantic 2.14.4, and attrs 26.1.0. Every output block is what the command preceding it printed. The three generators ran once, in generate.sh, and their Python packages are committed under generated/. Speakeasy was not run. Its options come from its documentation.

Footnotes

  1. Pydantic’s serialization page says so in one sentence, then adds that field values are left as is, so sub-models will not be converted to dictionaries. The same page documents model_dump() two sections earlier. pandas, offered an iterable of pairs per row, takes each pair for a cell, which is a reasonable reading of an unreasonable input. ↩︎ Back to text

  2. The generated OrderStatus subclasses str and Enum together. Python 3.11 added enum.StrEnum, whose __str__() is str.__str__(), so its members print as their values. OpenAPI Generator’s python page gives the generator’s language version as 3.11+, and the template writes the mixin form that predates it. pandas 3.0 infers dtype="str" for a collection of strings, and the text guide says that without PyArrow it is backed by a NumPy object array. An OrderStatus member is a str, so it goes in as it came, class and all. ↩︎ Back to text

  3. Speakeasy’s pagination page shows the loop as while response is not None, which is Python, and then says the next() function returns nil, nil when there are no more pages, which is Go. The page names no language and stands for every target that Speakeasy’s maturity table marks for pagination, so one sentence ended up in two. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.