The breaking-change check on your pull request is green, because the field you renamed was
optional in the old document. You deploy. Every client built from the last release reads
author as undefined, and a second change, a list that sorts newest first, breaks the
pagination code your largest customer wrote.
What you get
You will end up with a CI job that replays the previous release’s promises against a preview of the next one, and a matrix showing which check catches which change. This is for you if you deploy an API that generated clients already depend on.
Short answer
Check out the previous release tag in CI and point three checks at a preview deployment. Run Schemathesis with the previous spec, the SDK test corpus generated from that spec, and a hand-kept fixture set. A schema check misses a renamed optional field and a changed default sort order. The sdkgen corpus catches the rename; only a fixture that asserts the order catches the sort.
You will need
A tagged previous release of the OpenAPI document, a preview deployment per pull request, and
Node 22 or later for the sample. Comparing the two documents is
a separate check; this one
compares the old document with the new server. The sample’s previous.json is the Bookshelf
document as release 1.4.0 shipped it, with a templated server, http://{preview}/v1.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| Fern handwritten integration tests | Fern generates your SDKs and you keep integration tests beside them | Its generated wire tests run against a mock, so the live check is tests you write | You want a corpus generated for you |
| Hey API client with kept fixtures | A TypeScript or Python client and a fixture set you curate per release | Every expectation is written and updated by hand, release after release | You need coverage of every operation |
| Pact provider verification | Consumers you know record what they actually call | Each consumer has to adopt Pact, and nothing covers unrecorded calls | Your consumers are unknown |
| Schemathesis with the previous release’s spec | Broad generated coverage of every operation in the old document | It checks responses against schemas, so a valid response with changed behavior passes | You need to assert behavior |
| sdkgen live suite from the previous release | An sdkgen SDK, whose generated flow round-trips every entity operation | Its flow checks fields it wrote, not order, counts, or defaults | Your API is mostly reads with no writes allowed |
Voxgig maintains sdkgen. It is one of five options here, not the recommendation.
The trade-off is breadth against precision. Schemathesis and a generated corpus cover every operation without a person writing a case, and both stop at what a schema or a generic flow can state. Pact contracts and kept fixtures assert what real callers depend on, at the price of someone writing them.
Replay the previous corpus against each candidate
The SDK and its test suite were generated from previous.json. The suite has a live mode,
documented in the
run-a-live-suite how-to:
BOOKSHELF_TEST_LIVE=TRUE, the key in BOOKSHELF_APIKEY, and the host in
BOOKSHELF_SERVER_PREVIEW. corpus.mjs sets them and runs the suite as a child process:
const env = { ...process.env }
delete env.NODE_TEST_CONTEXT // inherited, it makes the inner node --test exit 0 on failure
Object.assign(env, { BOOKSHELF_TEST_LIVE: 'TRUE', BOOKSHELF_APIKEY: key, BOOKSHELF_SERVER_PREVIEW: host })
const child = spawn(process.execPath, ['--test', SUITE], { cwd: fileURLToPath(new URL('.', import.meta.url)), env })
preview.mjs stands in for five candidate releases. One is 1.4.0 unchanged, one adds an isbn
field, one renames author to writer, one lists newest first, and one returns published as a
string. replay.mjs runs three checks against each:
node replay.mjs
previous
1.4.0 response schema pass
1.4.0 sdkgen corpus pass
1.4.0 kept fixtures pass
records left 0
additive
1.4.0 response schema pass
1.4.0 sdkgen corpus pass
1.4.0 kept fixtures pass
records left 0
renamed
1.4.0 response schema pass
1.4.0 sdkgen corpus FAIL book.update.2 failed; book.load.3 failed
1.4.0 kept fixtures FAIL bk2 lacks author; bk3 lacks author; bk4 lacks author
records left 0
sorted
1.4.0 response schema pass
1.4.0 sdkgen corpus pass
1.4.0 kept fixtures FAIL list order fixture-3,fixture-2,fixture-1
records left 0
retyped
1.4.0 response schema FAIL bk2 published is not integer; bk3 published is not integer; bk4 published is not integer
1.4.0 sdkgen corpus pass
1.4.0 kept fixtures pass
records left 0
The schema row passes the rename. In 1.4.0 every Book property is optional, and
JSON Schema allows
properties the schema does not list, so { id, title, writer } is a valid 1.4.0 book. schema.mjs
applies exactly those rules.1 The corpus fails it, because its update step writes a mark into
author and its load step reads the mark back.
The sort change passes the corpus too. The generated flow checks that its record is in the list, not where.
Only the schema row fails the type change. The generated flow asserts the id at each step and
the mark it wrote into author, and the fixtures assert order, presence, and author. Neither
looks at the type of published.
Keep fixtures for behavior, with Hey API
Hey API generated a client from previous.json with one function per operation. It wires the
X-API-Key scheme to the auth option and leaves the base URL to you, since the document’s
server is templated. The fixtures are data, kept with the release, so a Python client from
@hey-api/openapi-python, Hey API’s HTTPX
generator, could read the same file:
"expect": {
"listOrder": ["fixture-1 Emma", "fixture-2 Ulysses", "fixture-3 Beloved"],
"loadedFields": ["author", "id", "published", "title"]
}
fixtures.mjs creates the three books through the client, compares the default list order and
each loaded record with the expectations, and deletes what it made in a finally. The order
assertion is the one no generated check on this page makes. Write one for every default a caller can
observe: sort order, page size, which fields appear when none are requested.
Run the same checks with Fern
Fern was not run here. Its testing page
describes generated unit tests that make no network calls. Its mock server tests, switched on by
generation.wireTests.enabled in sdk-config.yml, Fern’s current configuration format, run the SDK against a mock generated from
your definition. Neither reaches your preview. The page names handwritten integration tests as
the ones that run against your real server, and it marks the mock server tests and integration
tests as Enterprise plan features. Keep those tests at the release tag and run them as the
previous/ step in the next section runs the sdkgen corpus.
Check out the previous tag in CI
The job adds the previous release as a git worktree, writes its document to previous.json, and
runs each check against the preview for the pull request:
- name: Check out the previous release beside this one
run: |
TAG=$(git describe --tags --abbrev=0 --match 'v*')
git worktree add previous "$TAG"
git show "$TAG:openapi.json" > previous.json
fetch-depth: 0 on the checkout step is what lets
git describe find the tag. The runner image has no uv, so the job installs it with astral-sh/setup-uv. The Schemathesis step
then runs uvx schemathesis run previous.json --url with the key in --header, the forms its
quick start gives. A failing run looks
like the renamed block in the matrix: named steps, with the field each one lacked.
Check it worked
The tests pin the matrix. An additive release passes, and a rename fails the corpus and the fixtures but not the schema. A sort change fails only the fixtures, and a type change fails only the schema check. They also check cleanup, a refused key, and the workflow’s shape.
node --test previous.test.mjs | tail -8
# tests 8
# suites 0
# pass 8
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 44003.412364
When it goes wrong
A customer reports a changed order after a deploy that passed every check, because no check
asserted the default. Add the assertion to fixtures.json at the release that introduced the default, so
the next release is checked against it.
The corpus passes against a preview that refuses the key. A nested node --test that inherits
NODE_TEST_CONTEXT reports to its parent and exits 0. Remove the variable before spawning, as
corpus.mjs does; one test runs the corpus with a stale key and expects a failure.
The schema row passes a response with a wrong type. A validator that takes the schema as a
second parameter was handed straight to flatMap, which passes the index there, so every record
was checked against nothing. Wrap it in a one-argument function, as fixtures.mjs does; one test
changes a type and expects the row to fail.
The preview fills with test books. A check stopped between its create and its delete. Delete in
a finally, as fixtures.mjs does, and give the preview a database that is thrown away with it.
When not to do this
Do not run this against production. The corpus and the fixtures both write, and a replay that dies midway leaves records behind.
Do not treat a green matrix as a compatibility promise. Its coverage stops at the operations the corpus generates and the defaults someone wrote down.
Do not adopt sdkgen to get the corpus. If your SDKs come from Fern, its integration tests kept at the release tag do the same job with the generator you already run. With one TypeScript client from Hey API, a fixture file like this one is less machinery than a second generator. It is also the only check on this page that sees order.
Related how-tos
Last verified
Verified 2026-11-20 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3,
@hey-api/openapi-ts 0.99.0 and TypeScript 5.9.3. Every output block is what the command preceding it printed. Hey API was run once and its compiled output committed. Fern was not run. Its options come from its documentation. Schemathesis and Pact were not run, and the workflow was
not run on GitHub.
Footnotes
-
The object reference says it in two sentences: properties defined by
propertiesare not required by default, and by default any additional properties are allowed. A schema written withoutrequireddescribes every object that has the right types where it has anything, which includes an empty one. ↩︎ Back to text