The agent’s pull request adds exponential backoff, and every test passes against a mock that always answers 200. The retry loop has never run. When the API returns 503 for an hour, the client retries with no cap and the job hangs, because no test ever reached the code that gives up.
What you get
You will end up with a test that feeds the agent’s client scripted failures and asserts how many calls it made and how long it would have waited. This is for you if a coding agent wrote retry code you are about to merge.
Short answer
Inject failures beneath the agent’s retry code, at the fetch it calls. Use a scripted fetch
in process, a local server that drops sockets, Toxiproxy for TCP faults, or sdkgen’s netsim
feature inside a generated SDK. Pass the client a wait that records instead of sleeping. Then
assert one retry on a 503, none on a 400, and a bounded give-up on a permanent outage.
You will need
Node 22 or later, a client with retry code written by a coding agent, and an OpenAPI 3 document
for the API it calls. Go 1.24 or later for the Go part, which uses the standard library only.
Verified 2026-11-20 against Node 22.23.5 and Go 1.24.7. The sample’s agent-written client
wraps openapi-fetch, whose fetch option replaces
globalThis.fetch, and that option is where the faults go.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| openapi-typescript and openapi-fetch with a scripted fetch | The agent wrapped openapi-fetch, and the fault belongs in process | A fake fetch you script per test, and no wire-level behavior unless you add a server | You need resets and latency on a real socket |
| sdkgen netsim feature | The client is an sdkgen SDK with the retry feature | Exists only inside a generated SDK, so it tests sdkgen’s retry rather than the agent’s | The agent wrote the retry code itself |
| Swagger Codegen fetch argument | A typescript-fetch client, where the constructor’s third argument is the fetch | No retry in the generated client, so the wrapper and its faults are both yours | You want retry generated rather than written |
| Toxiproxy | Any client in any language, and the faults are latency, resets, and timeouts | A proxy process in the test setup, and no way to answer with an HTTP status | The assertion is about 503 versus 400 |
| WireMock fault and delay stubs | Declarative HTTP faults: statuses, delays, and connection resets per stub | A JVM or container beside the tests, and faults only on requests it serves itself | Tests should run in the client’s own process |
Voxgig maintains sdkgen. It is one of five options here, not the recommendation.
Toxiproxy reaches every client because it sits on the TCP connection, but it cannot send a 503.
WireMock sends any status and delay from a stub file, and needs a JVM to do it. A scripted fetch
costs no process at all and only covers clients that take a fetch. netsim is deterministic and
in process, but it tests sdkgen’s retry, not the code the agent wrote.
Put the faults beneath the retry code
The agent’s client passes retryingFetch(inner) as
openapi-fetch’s fetch. A fault must sit in inner, under the loop, so every failure passes
through the retry decision. faults.mjs provides two injectors with one script format: a
status per call, reset for a dropped connection, and the last entry repeating.
// One entry per call: a status, 'reset' for a dropped connection, or 'ok'.
// After the script runs out, the last entry repeats, so ['503'] is an outage.
export function scriptedFetch(script, body = { id: 'b1', title: 'Dune' }) {
const calls = []
const fetch = async (input, init) => {
const request = new Request(input, init)
const step = script[Math.min(calls.length, script.length - 1)]
calls.push(`${request.method} ${new URL(request.url).pathname} -> ${step}`)
if (step === 'reset') throw new TypeError('fetch failed', { cause: { code: 'ECONNRESET' } })
faultServer serves the same script over TCP on 127.0.0.1 and destroys the socket on reset,
so Node’s real fetch sees a dropped connection. That is the fault Toxiproxy’s reset_peer
toxic produces.1 The wait comes from the client’s sleep option, which the demo replaces with
a function that records each delay.
node --experimental-strip-types demo.mjs
503 then ok (in-process)
calls 2, waits [100], result 200 Dune
400 (in-process)
calls 1, waits [], result 400
503 for good (in-process)
calls 3, waits [100,200], result 503
reset then ok (tcp)
calls 2, waits [100], result 200 Dune
reset for good (tcp)
calls 3, waits [100,200], result threw fetch failed
A 503 costs one retry and a 100 ms wait. The 400 comes back at once, because RFC 9110 frames a 400 as a client error.2 A permanent 503 stops after three calls. The demo finishes in milliseconds, since no wait was slept.
Run the same faults through sdkgen’s netsim feature
In an sdkgen SDK the injector is a feature. netsim sits beneath retry in the transport chain,
and test answers from a seed, so the whole run stays in process. Add all three features when
you scaffold. generate.sh ran these commands once and kept the compiled .js files in
generated/:
npx --yes @voxgig/create-sdkgen@0.31.9 bookshelf --def api.json --target ts \
--feature retry --feature netsim --feature test
(cd bookshelf-sdk/.sdk && npm run generate)
(cd bookshelf-sdk/ts && npm install --no-audit --no-fund && npm run build)
Each feature is off until its options say active: true. netsim.mjs turns on all three per
client, passes the fault as netsim options, and gives retry a sleep that records:
export function client(netsim, sleeps) {
return new BookshelfSDK({
feature: {
test: { active: true, entity: { book: { b1: { id: 'b1', title: 'Dune' } } } },
netsim: { active: true, ...netsim },
retry: { active: true, retries: 2, minDelay: 100, jitter: false, sleep: (ms) => { sleeps.push(ms) } },
},
})
}
The four cases pass { failTimes: 1, failStatus: 503 }, the same with 400, { errorTimes: 1 }
for a connection error, and { offline: true }:
node netsim.mjs
503 then ok calls 2, waits [100], ok Dune
400 calls 1, waits [], threw BookshelfSDK: load: request: 400: Simulated Failure
reset then ok calls 2, waits [100], ok Dune
offline for good calls 3, waits [100,200], threw BookshelfSDK: load: Simulated network offline
The numbers match the agent’s client because the options were chosen to match. client._retry
counts retries rather than calls, so attempts: 1 means two calls. The run tests sdkgen’s retry
feature, which matters when sdkgen generated the SDK. It says nothing about retry code the agent
wrote.
Inject the same faults with openapi-fetch and Swagger Codegen
openapi-fetch 0.17.0 has no retry, and openapi-typescript 7.13.0 writes types only, so the
agent’s wrapper is the retry and the scripted fetch is the fault. Inject through the fetch
option, not through a middleware. A middleware onRequest that returns a Response skips the
request, which the middleware page
documents. A 503 returned there never passes through the retry wrapper, as the pitfall below
shows. This pair beats sdkgen when the client must stay a thin typed layer over fetch with no
generated runtime.
Swagger Codegen 3.0.85’s typescript-fetch client has no retry: its constructor is
constructor(configuration?, basePath = BASE_PATH, fetch = isomorphicFetch). Wrap the third
argument with the same retryingFetch, and put the faults under it.
node --experimental-strip-types swagger.mjs
generated client, 503 then ok calls 1, waits [], rejected with a Response 503
wrapped, 503 then ok calls 2, waits [100], resolved Dune
wrapped, 400 calls 1, waits [], rejected with a Response 400
The generated client rejects with the Response itself, not an Error. It beats sdkgen for a
team that ships ten languages from one Java tool and already has a retry policy per language.
Inject the faults beneath a Go transport
The leaf covers Go too, and a Go client takes its faults the same way, one layer down. The agent’s
retry code is an http.RoundTripper whose Next is the injector and whose Sleep records:
func (t *Retry) RoundTrip(req *http.Request) (*http.Response, error) {
if !slices.Contains(idempotent, req.Method) {
return t.Next.RoundTrip(req)
}
for attempt := 0; ; attempt++ {
resp, err := t.Next.RoundTrip(req)
transient := err != nil || slices.Contains(t.Statuses, resp.StatusCode)
if !transient || attempt >= t.Retries {
return resp, err
}
if resp != nil {
io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}
if err := t.Sleep(req.Context(), min(t.MaxDelay, t.BaseDelay<<attempt)); err != nil {
return nil, err
}
}
}
go/faults.go holds the Go twin of scriptedFetch, a Script transport that returns a
syscall.ECONNRESET error for reset. Run the cases:
go run -C go .
503 then ok calls 2, waits [100ms], 200 {"id":"b1","title":"Dune"}
400 calls 1, waits [], 400 {"error":"status 400"}
503 for good calls 3, waits [100ms 200ms], 503 {"error":"status 503"}
reset then ok calls 2, waits [100ms], 200 {"id":"b1","title":"Dune"}
reset for good calls 3, waits [100ms 200ms], error Get "http://bookshelf.test/v1/books/b1": read tcp: connection reset by peer
POST, 503 calls 1, waits [], 503 {"error":"status 503"}
The POST goes out once, because RFC 9110 does not count POST among the
idempotent methods. client.ts
applies the same list. The Go part has no sdkgen run: the sdkgen sample on this page is the TypeScript
target only.
Check it worked
Ten tests cover both injectors, the give-up path, the POST that is not retried, the middleware
trap, netsim, Swagger Codegen, and the Go transport, whose own go test suite the last test
runs. The first asserts that the recorded 100 ms wait was not slept.
node --experimental-strip-types --test retry.test.mjs
1..10
# tests 10
# suites 0
# pass 10
# fail 0
# cancelled 0
# skipped 0
# todo 0
When it goes wrong
Only the happy retry was tested, and the client retries forever on an outage. pitfall.mjs holds
a loop that retries every 5xx with no cap. Feed it a permanent 503 with a wait that stops the
demo:
node --experimental-strip-types pitfall.mjs
naive retry, permanent 503
calls 50, result: stopped after 50 waits
the first 20 waits alone add up to 29 hours
capped retry, permanent 503
calls 3, waits [100,200], result: 503
503 from a middleware, above the retry
transport calls 0, waits [], result: 503
The uncapped loop has no exit and doubles its wait unbounded. Add a retry count and a maxDelay,
and keep a test with a permanent outage in it.
The fault never reaches the retry code. It was injected outside the wrapper, in a middleware, so the
transport made zero calls and nothing retried. Move the injector into the fetch the wrapper
calls.
The suite takes minutes. A test exercises real waits. Pass sleep in every test, and assert on
the recorded delays.
When not to do this
Do not use sdkgen’s netsim to verify retry code the agent wrote by hand. It exercises the
retry feature sdkgen generated, so the agent’s loop is not under test. Use a scripted fetch or a
fault server, and keep netsim for an SDK that sdkgen generated.
Do not reach for Toxiproxy to test status-code decisions. It works on TCP, so a 503 against a 400 is outside its reach. Pair it with an HTTP-level injector, or use WireMock, which does both.
Do not retry a POST because a test showed the retry works. A retried write that the server
committed is a second record. Scope the retry to idempotent methods, as client.ts and the Go
transport do, or add an idempotency key first.
Do not trust a retry test without a give-up case. One permanent outage per suite is the test that catches the loop with no exit.
Related how-tos
Last verified
Verified 2026-11-20 against Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3,
openapi-fetch 0.17.0, openapi-typescript 7.13.0 and Swagger Codegen 3.0.85, compiled with
TypeScript 5.9.3, and Go 1.24.7 for the Go part. Every output block is what the command preceding
it printed. Swagger Codegen
was run once and its compiled output is committed. Toxiproxy and WireMock were not run; their
behavior comes from their documentation.
Footnotes
-
The Toxiproxy README describes
reset_peeras a simulated TCP RESET, and says the tool has been in all development and test environments at Shopify since October 2014. Its README lists nine entries under toxics, fromlatencytopacket_loss, each a property of the connection rather than of HTTP. One of the nine,down, the README itself says is not technically a toxic. ↩︎ Back to text -
RFC 9110 describes a 503 as a temporary overload or scheduled maintenance that will likely pass after some delay, and lets the server say how long in
Retry-After. A 400 is a request perceived as a client error. Retrying one is sending the same mistake again. ↩︎ Back to text