The SDK’s tests pass against the mock on every pull request. The sandbox API has renamed a field
since the mock was recorded, so the mock still answers the shape the tests expect. Nothing runs
against the sandbox before release, so the first customer to call list gets a type error.
What you get
You will end up with a nightly workflow that runs the SDK’s integration suite against a sandbox tenant, and a release job that refuses to publish without a recent green run. This is for you if you publish an SDK and only mocks test it before release.
Short answer
Schedule a GitHub Actions workflow that runs the SDK’s integration suite against a sandbox
tenant, with the key from the CI secret store and a sweep step that deletes tagged records. In
the release workflow, read the runs with gh run list and publish only when the newest finished
run succeeded within 24 hours. An sdkgen SDK reruns its mock corpus live when
BOOKSHELF_TEST_LIVE=TRUE.
You will need
An SDK with an integration test suite, a sandbox tenant, and its key stored as a
GitHub Actions secret.
Node 22 or later runs the sample. The sample’s api.json is the five-operation Bookshelf
document with a templated server, http://{sandbox}/v1, and an X-API-Key scheme. A stand-in
on 127.0.0.1 plays the sandbox tenant.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
Hand-written smoke tests in a workflow_dispatch job | A handful of calls you choose, run by hand before a release | Coverage stops at the calls someone remembered to write, and nobody runs a manual job nightly | You want the gate to need no person |
| NSwag client with smoke checks | A TypeScript or C# client generated by NSwag | NSwag writes no tests, so every live check is code you maintain | Your generator ships a test suite |
| Schemathesis on a schedule | You want drift between the spec and the deployed server | It tests the server against the spec, so a bug inside the SDK passes | The SDK’s own code is what you need to prove |
| sdkgen live suite | An sdkgen SDK, and the same tests offline and live | A failed delete inside the suite leaves the record it created behind | Your sandbox cannot take writes |
| Swagger Codegen test stubs | A Swagger Codegen client and a test file per API to fill in | Every stub body is pass, so the suite proves nothing until written | You need coverage on day one |
Voxgig maintains sdkgen. It is one of five options here, not the recommendation.
The scheduled workflow and the release gate are the same whichever suite runs. What changes is what the suite covers. Reusing a generated corpus is coverage you did not write, at the price of fixtures you did not design. Schemathesis finds spec-versus-server drift but not SDK bugs.
Run the generated corpus against the sandbox
The generated entity test is one test with two modes. It reads BOOKSHELF_TEST_LIVE and, when
it is TRUE, sends the same flow to the server instead of the in-memory mock:
(0, node_test_1.test)('basic', async (t) => {
const live = 'TRUE' === process.env.BOOKSHELF_TEST_LIVE;
for (const op of ['create', 'list', 'update', 'load', 'remove']) {
if (!live && (0, utility_1.maybeSkipControl)(t, 'entityOp', 'book.' + op, live))
return;
}
const setup = basicSetup();
Offline, it runs against fixtures with fixed ids such as BOOK00:
node --test generated/dist-test/entity/book/BookEntity.test.js | tail -8
# tests 6
# suites 1
# pass 5
# fail 0
# cancelled 0
# skipped 1
# todo 0
# duration_ms 153.118268
Live, the run-a-live-suite how-to
names the three inputs: BOOKSHELF_TEST_LIVE=TRUE, the key in BOOKSHELF_APIKEY, and each
server variable as BOOKSHELF_SERVER_<NAME>. live.mjs starts the stand-in, sets all three,
and runs the suite twice:
node live.mjs
CI secret matches the sandbox key
live steps: 6 planned, 6 passed, 0 failed, 0 blocked
suite exit: 0, green
requests: POST /v1/books, GET /v1/books, PATCH /v1/books/bk1, GET /v1/books/bk1, DELETE /v1/books/bk1, GET /v1/books
records left in the sandbox: 0
sandbox key rotated, CI secret not updated
live steps: 6 planned, 0 passed, 3 failed, 3 blocked
suite exit: 1, red
requests: POST /v1/books, GET /v1/books, GET /v1/books
records left in the sandbox: 0
The live flow used bk1, the id the server returned, rather than a fixture id. It deleted the
record it made and then listed again to confirm the delete.
Schedule it and gate the release on it
The nightly workflow runs the suite with the secret and a repository variable for the host. Its
sweep step runs with if: always(), so it runs after a red suite too:
on:
schedule:
- cron: '17 3 * * *'
workflow_dispatch:
Pick a minute other than zero; GitHub documents delays at the start of every hour.1
The release workflow asks GitHub for the last runs and hands them to gate.mjs. The job
declares actions: read, the permission the
workflow runs API asks for, which
gh run list calls:
run: |
gh run list --workflow live-suite.yml --branch main --limit 20 \
--json status,conclusion,startedAt,event > runs.json
node gate.mjs runs.json
The gate takes the newest finished run that was not cancelled or skipped, not the newest green one. A key rotated after a green run makes the next run red, and that red run is what blocks:
node gate.mjs runs/rotated.json 2026-11-20T09:00:00Z; echo "exit $?"
release blocked: newest live run is failure, 1.3 h ago
exit 1
runs/rotated.json holds a green nightly from 5.7 hours earlier. A rule of “any green run in the
past 24 hours” would have published.
Write the live checks for Swagger Codegen and NSwag
Swagger Codegen writes test files you fill in. Its Python client came with
test/test_default_api.py, one method per operation, each with a body of pass, so the
scheduled job would be green from the first night:
def test_list_books(self):
"""Test case for list_books
"""
pass
Fill each stub with a call against configuration.host, which the client defaults to the
document’s literal http://{sandbox}/v1. Run it as java -DapiTests=false -jar ... if you would
rather not ship empty tests; the stub file is then left out.2
NSwag writes no tests and, for this document, no code for the X-API-Key scheme. Its TypeScript
constructor takes a base URL and an object with a fetch, which is where the key goes:
const fetchWithKey = (url, init = {}) => fetch(url, { ...init, headers: { ...init.headers, 'X-API-Key': key } })
const client = new Client(`http://${host}/v1`, { fetch: fetchWithKey })
smoke.mjs is the hand-written smoke list over that client. It tags what it creates with
live- and deletes it in a finally:
node smoke.mjs
list books: ok
create a tagged book: ok
load it back: ok
delete it: ok
records left: 0
Check it worked
Rotate the sandbox key without updating the secret, start the live workflow by hand, then push a release tag. The tests run that sequence against the stand-in. They also check the gate’s edges, the sweep, and the workflow shape:
node --test live.test.mjs | tail -8
# tests 11
# suites 0
# pass 11
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 21511.298524
live.mjs removes NODE_TEST_CONTEXT before it starts the inner node --test. With it
inherited, the inner run reports to the outer one and exits 0 on a red suite.
When it goes wrong
The sandbox hits its record quota within a week. A delete failed inside the suite, and the
suite does not retry it. cleanup.mjs makes the sandbox answer one DELETE with 500 for each suite:
node cleanup.mjs
generated live suite: red, POST bodies [{}]
left after the suite: bk1 tagged=true
sweep deleted 1, left after the sweep: none
smoke checks: delete it: 500
left after the suite: bk1 tagged=true
sweep deleted 1, left after the sweep: none
The generated flow builds its create body from the request schema’s required fields and examples.
Bookshelf’s Book declares neither, so the POST carried {}. The update step writes the
Mark01- value into author. A failed update alone does no harm, because the suite still runs its
delete. A record whose update and delete both fail carries no mark, so give the suite a tenant of
its own, or have the sweep delete everything in it.
The gate blocks every release, and the newest live run is weeks old. In a public repository GitHub disables a scheduled workflow after 60 days without repository activity, so no run is newer than the day it stopped. Re-enable the workflow; the gate failing closed is the intended behavior.
When not to do this
Do not point a live suite at production. The generated flow creates, updates, and deletes, and a
failed delete leaves a record behind, as cleanup.mjs shows.
Do not gate on a run that tested other code. A nightly run checks main as it stood that night, so a release cut from a later commit needs a manual run first.
Do not adopt sdkgen for its live suite alone. If your SDK comes from Swagger Codegen, filling its stubs keeps one generator and one test layout; with NSwag, a dozen smoke checks over its client cover the calls customers make. Run Schemathesis beside either when the risk is the server drifting from the spec.
Related how-tos
Last verified
Verified 2026-11-20 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3,
NSwag 14.8.0 and Swagger Codegen 3.0.85. Every output block is what the command preceding it printed. NSwag and Swagger Codegen were run once on api.json and their output committed. The
two workflows were not run on GitHub; a test checks their shape. Schemathesis was not run.
Footnotes
-
GitHub’s schedule event notes that runs can be delayed during periods of high load, and names the start of every hour as such a period. It adds that some queued jobs may be dropped if the load is high enough. Every schedule written
0 * * * *makes the hour a little busier. ↩︎ Back to text -
Swagger Codegen’s selective generation page describes
-DapiTestsand-DmodelTestsas options that default to true. They are Java system properties rather than generator flags, which is why they go before-jar. The generated class names itself in its docstring asDefaultApi unit test stubs. ↩︎ Back to text