# How to test an API release against the spec it replaces

> Replay what the last release promised against a preview of the next one, and catch the behavior changes that a schema diff and a schema check both pass.

Source: https://voxgig.com/howto/test-a-release-against-the-previous-spec

- Audience: platform-team
- Level: advanced
- Languages: python, typescript
- Verified: 2026-11-20
- Published: 2026-11-20

## Short answer

Check out the previous release tag in CI and point three checks at a preview deployment. Run Schemathesis with the previous spec, the SDK test corpus generated from that spec, and a hand-kept fixture set. A schema check misses a renamed optional field and a changed default sort order. The sdkgen corpus catches the rename; only a fixture that asserts the order catches the sort.

---
## You will need

A tagged previous release of the OpenAPI document, a preview deployment per pull request, and
Node 22 or later for the sample. Comparing the two documents is
[a separate check](/howto/detect-breaking-changes-between-openapi-versions-in-ci); this one
compares the old document with the new server. The sample's `previous.json` is the Bookshelf
document as release 1.4.0 shipped it, with a templated server, `http://{preview}/v1`.

## Approaches compared

| Approach | When it fits | What it costs you | When to pick something else |
| --- | --- | --- | --- |
| [Fern handwritten integration tests](https://buildwithfern.com/learn/sdks/current/deep-dives/testing) | Fern generates your SDKs and you keep integration tests beside them | Its generated wire tests run against a mock, so the live check is tests you write | You want a corpus generated for you |
| [Hey API client with kept fixtures](https://heyapi.dev/docs/openapi/typescript/clients/fetch) | A TypeScript or Python client and a fixture set you curate per release | Every expectation is written and updated by hand, release after release | You need coverage of every operation |
| [Pact provider verification](https://docs.pact.io/getting_started/provider_verification) | Consumers you know record what they actually call | Each consumer has to adopt Pact, and nothing covers unrecorded calls | Your consumers are unknown |
| [Schemathesis with the previous release's spec](https://schemathesis.readthedocs.io/en/stable/quick-start/) | Broad generated coverage of every operation in the old document | It checks responses against schemas, so a valid response with changed behavior passes | You need to assert behavior |
| [sdkgen live suite from the previous release](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/run-a-live-suite.md) | An sdkgen SDK, whose generated flow round-trips every entity operation | Its flow checks fields it wrote, not order, counts, or defaults | Your API is mostly reads with no writes allowed |

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

The trade-off is breadth against precision. Schemathesis and a generated corpus cover every
operation without a person writing a case, and both stop at what a schema or a generic flow can
state. Pact contracts and kept fixtures assert what real callers depend on, at the price of
someone writing them.

## Replay the previous corpus against each candidate

The SDK and its test suite were generated from `previous.json`. The suite has a live mode,
documented in the
[run-a-live-suite how-to](https://github.com/voxgig/sdkgen/blob/main/docs/how-to/run-a-live-suite.md):
`BOOKSHELF_TEST_LIVE=TRUE`, the key in `BOOKSHELF_APIKEY`, and the host in
`BOOKSHELF_SERVER_PREVIEW`. `corpus.mjs` sets them and runs the suite as a child process:

```js title="corpus.mjs"
  const env = { ...process.env }
  delete env.NODE_TEST_CONTEXT // inherited, it makes the inner node --test exit 0 on failure
  Object.assign(env, { BOOKSHELF_TEST_LIVE: 'TRUE', BOOKSHELF_APIKEY: key, BOOKSHELF_SERVER_PREVIEW: host })
  const child = spawn(process.execPath, ['--test', SUITE], { cwd: fileURLToPath(new URL('.', import.meta.url)), env })
```

`preview.mjs` stands in for five candidate releases. One is 1.4.0 unchanged, one adds an `isbn`
field, one renames `author` to `writer`, one lists newest first, and one returns `published` as a
string. `replay.mjs` runs three checks against each:

```bash
node replay.mjs
```

```text output
previous
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0
additive
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0
renamed
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     FAIL book.update.2 failed; book.load.3 failed
  1.4.0 kept fixtures     FAIL bk2 lacks author; bk3 lacks author; bk4 lacks author
  records left            0
sorted
  1.4.0 response schema   pass
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     FAIL list order fixture-3,fixture-2,fixture-1
  records left            0
retyped
  1.4.0 response schema   FAIL bk2 published is not integer; bk3 published is not integer; bk4 published is not integer
  1.4.0 sdkgen corpus     pass
  1.4.0 kept fixtures     pass
  records left            0
```

The schema row passes the rename. In 1.4.0 every `Book` property is optional, and
[JSON Schema](https://json-schema.org/understanding-json-schema/reference/object) allows
properties the schema does not list, so `{ id, title, writer }` is a valid 1.4.0 book. `schema.mjs`
applies exactly those rules.[^1] The corpus fails it, because its update step writes a mark into
`author` and its load step reads the mark back.

The sort change passes the corpus too. The generated flow checks that its record is in the list,
not where.

Only the schema row fails the type change. The generated flow asserts the `id` at each step and
the mark it wrote into `author`, and the fixtures assert order, presence, and `author`. Neither
looks at the type of `published`.

## Keep fixtures for behavior, with Hey API

Hey API generated a client from `previous.json` with one function per operation. It wires the
`X-API-Key` scheme to the `auth` option and leaves the base URL to you, since the document's
server is templated. The fixtures are data, kept with the release, so a Python client from
[`@hey-api/openapi-python`](https://heyapi.dev/docs/openapi/python/get-started), Hey API's HTTPX
generator, could read the same file:

```json title="fixtures.json"
  "expect": {
    "listOrder": ["fixture-1 Emma", "fixture-2 Ulysses", "fixture-3 Beloved"],
    "loadedFields": ["author", "id", "published", "title"]
  }
```

`fixtures.mjs` creates the three books through the client, compares the default list order and
each loaded record with the expectations, and deletes what it made in a `finally`. The order
assertion is the one no generated check on this page makes. Write one for every default a caller can
observe: sort order, page size, which fields appear when none are requested.

## Run the same checks with Fern

Fern was not run here. Its [testing page](https://buildwithfern.com/learn/sdks/current/deep-dives/testing)
describes generated unit tests that make no network calls. Its mock server tests, switched on by
`generation.wireTests.enabled` in `sdk-config.yml`, Fern's current configuration format, run the SDK against a mock generated from
your definition. Neither reaches your preview. The page names handwritten integration tests as
the ones that run against your real server, and it marks the mock server tests and integration
tests as Enterprise plan features. Keep those tests at the release tag and run them as the
`previous/` step in the next section runs the sdkgen corpus.

## Check out the previous tag in CI

The job adds the previous release as a `git worktree`, writes its document to `previous.json`, and
runs each check against the preview for the pull request:

```yaml title="workflows/previous-release.yml"
      - name: Check out the previous release beside this one
        run: |
          TAG=$(git describe --tags --abbrev=0 --match 'v*')
          git worktree add previous "$TAG"
          git show "$TAG:openapi.json" > previous.json
```

`fetch-depth: 0` on the checkout step is what lets
[`git describe`](https://git-scm.com/docs/git-describe) find the tag. The runner image has no uv, so the job installs it with `astral-sh/setup-uv`. The Schemathesis step
then runs `uvx schemathesis run previous.json --url` with the key in `--header`, the forms its
quick start gives. A failing run looks
like the `renamed` block in the matrix: named steps, with the field each one lacked.

## Check it worked

The tests pin the matrix. An additive release passes, and a rename fails the corpus and the
fixtures but not the schema. A sort change fails only the fixtures, and a type change fails only
the schema check. They also check cleanup, a refused key, and the workflow's shape.

```bash
node --test previous.test.mjs | tail -8
```

```text output
# tests 8
# suites 0
# pass 8
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 44003.412364
```

## When it goes wrong

A customer reports a changed order after a deploy that passed every check, because no check
asserted the default. Add the assertion to `fixtures.json` at the release that introduced the default, so
the next release is checked against it.

The corpus passes against a preview that refuses the key. A nested `node --test` that inherits
`NODE_TEST_CONTEXT` reports to its parent and exits 0. Remove the variable before spawning, as
`corpus.mjs` does; one test runs the corpus with a stale key and expects a failure.

The schema row passes a response with a wrong type. A validator that takes the schema as a
second parameter was handed straight to `flatMap`, which passes the index there, so every record
was checked against nothing. Wrap it in a one-argument function, as `fixtures.mjs` does; one test
changes a type and expects the row to fail.

The preview fills with test books. A check stopped between its create and its delete. Delete in
a `finally`, as `fixtures.mjs` does, and give the preview a database that is thrown away with it.

## When not to do this

Do not run this against production. The corpus and the fixtures both write, and a replay that
dies midway leaves records behind.

Do not treat a green matrix as a compatibility promise. Its coverage stops at the operations
the corpus generates and the defaults someone wrote down.

Do not adopt sdkgen to get the corpus. If your SDKs come from Fern, its integration tests kept at
the release tag do the same job with the generator you already run. With one TypeScript client from
Hey API, a fixture file like this one is less machinery than a second generator. It is also the
only check on this page that sees order.

## Related how-tos

- [Detect breaking changes between two OpenAPI versions in CI](/howto/detect-breaking-changes-between-openapi-versions-in-ci)

- [Choose between a shared test corpus and Pact contracts](/howto/choose-between-a-shared-corpus-and-pact-contracts)

- [Choose consumer-driven or spec-driven contract tests](/howto/choose-consumer-driven-or-spec-driven-contract-tests)

- [Layer unit, integration, and live tests for an SDK](/howto/layer-unit-integration-and-live-sdk-tests)

- [Run a scheduled live suite that gates an SDK release](/howto/scheduled-live-suite-before-an-sdk-release)

## Last verified

Verified 2026-11-20 against Node 22.23.5, `@voxgig/create-sdkgen` 0.31.9, `@voxgig/sdkgen` 4.44.3,
`@hey-api/openapi-ts` 0.99.0 and TypeScript 5.9.3. Every output block is what the command preceding it printed. Hey API was run once and its compiled output committed. Fern was not run. Its options come from its documentation. Schemathesis and Pact were not run, and the workflow was
not run on GitHub.

[^1]: The [object reference](https://json-schema.org/understanding-json-schema/reference/object)
says it in two sentences: properties defined by `properties` are not required by default, and by
default any additional properties are allowed. A schema written without `required` describes
every object that has the right types where it has anything, which includes an empty one.