How-to › Use AI to do the integration

How to set a per-task token budget that stops a runaway agent#

Give each agent task a token allowance, stop between model calls when it is gone, and return what the task found with the reason it stopped.

Audience
Agent builder
Level
intermediate
Topic
Evaluate and observe agents
Languages
Python and TypeScript
Verified

Read first: Account for tokens and cost per agent task

An agent asked to find one parcel searches, gets nothing, and searches again with the name reworded. Every call sends the whole transcript again, so every call costs more than the last. When its step limit ends the run, it has made 25 model calls and spent 52,405 tokens on a question that takes 248 when it can be answered.

What you get

You will end up with an agent loop that gives each task a token allowance, stops between calls when it is gone, and returns what it found with the reason. This is for you if a looping agent can spend money nobody budgeted.

Short answer

Give each agent task a token allowance by building one sdkgen client per task, with the cost feature reading usage.total_tokens and its budget set to the allowance. The client refuses the next model call with cost_budget once the allowance is spent, and the loop turns that refusal into a partial result with a reason. The check runs between calls, so hold back one call’s worth of tokens, and keep a second allowance per tenant.

You will need

Node 22 or later, Python 3 for the loop with no SDK, and a model API whose responses report token usage. Verified 2026-11-19 against Node 22.23.5, Python 3.13.18, @voxgig/create-sdkgen 0.31.9, and @voxgig/sdkgen 4.44.3. Reading and pricing the usage fields is covered in account for tokens and cost per agent task.

The model here is a stand-in on the loopback interface. Its usage object has the fields the Responses API documents, and its counts are a simplification: one input token per four characters of input.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
Fern custom fetcherAn SDK you generate with Fern, whose users should bring their own request policyA producer setting, allowCustomFetcher, plus a wrapper each user writes to count and refuseThe SDK’s producer left the option off
LangGraph recursion limitA graph that must not run forever, whatever a step costsIt counts super-steps, defaults to 10,007 of them, and ends the run with an exceptionThe cost of a step grows with the transcript
LiteLLM budgetsMany services behind one gateway, capped per key, team, or customerA proxy and a database to run, budgets in dollars, and no view of the taskThe limit belongs to one task in one process
OpenAI Agents SDK max_turnsAn agent on that SDK that needs a ceiling on turnsIt counts turns, not tokens, and ends in MaxTurnsExceeded unless an error_handlers entry supplies a final outputTurns differ widely in cost
sdkgen cost feature budgetA model API client generated by sdkgen, built once per taskA cutoff: the last admitted call can pass the allowance, and each client counts aloneThe model SDK is not sdkgen’s, where a counter in the loop does the same
Speakeasy SDK hooksAn SDK you generate with Speakeasy on a Business or Enterprise planHooks are compiled into the SDK by its producer, and limited to those plansYou only consume the SDK and cannot add a hook

Voxgig maintains sdkgen. It is one of six options here, not the recommendation.

Framework limits count steps, and a step is not a unit of cost: the last call of the run below costs 71 times the first. A proxy such as LiteLLM can refuse a call before it is sent, but it budgets keys, teams, and customers, not tasks. A cutoff between calls needs no framework and no proxy, but it lets the last call through.

Give each task its own client and allowance

The cost feature has to be in the generated client, so name it when you scaffold.

npx --yes @voxgig/create-sdkgen@0.31.9 quill --def api.json --target ts --feature cost
cd quill-sdk/.sdk
npm run generate

Then build one client per task. path points at the usage figure in the response body, perUnit prices one token as one unit, and budget with onBudget: 'deny' is the allowance.

  const client = new QuillSDK({
    base,
    feature: {
      cost: {
        active: true,
        currency: 'tokens',
        // Read each call's price from the response body, one unit per token.
        path: 'usage.total_tokens',
        perUnit: 1,
        // A response with no usage counts as a worst-case call, not as nothing.
        unit: WORST_CALL,
        // The budget is checked against tokens already spent, so the last call
        // admitted can pass it. Hold back one call's worth to end under the allowance.
        ...(limited ? { budget: allowance - holdBack, onBudget: 'deny' } : {}),
        // Called after every model call with the tokens it was charged.
        sink: (record) => {
          costs.push(record.amount)
          tenant?.add(record.amount)
        },
      },
    },
  })

The budget belongs to the client instance, which is what makes it per task. sdkgen’s reference describes it as a cutoff scoped to one client, not a hard cap.1 currency is only a label, so the feature that counts dollars elsewhere counts tokens here. After every call client._cost.budget reports the limit, the tokens spent, and what remains.

Turn the refusal into a partial result

Once the allowance is spent, the client refuses the next call with cost_budget before sending it. The loop catches that one code and returns, and lets every other error through. It keeps its step limit as a second stop, and every exit goes through one function that reports the status, the reason, the tokens, and the tool results so far.

  for (let step = 1; step <= maxSteps; step++) {
    if (tenant?.spent) return stop('stopped', 'tenant_budget')
    let reply
    try {
      reply = (await client.Response().create({ model: 'quill-small', input, max_output_tokens: 256 })).data()
    } catch (err) {
      // The client refused to start the call: the allowance is gone. Anything else is a real failure.
      if (err.code === 'cost_budget') return stop('stopped', 'token_budget')
      throw err
    }
    if (!reply.tool_call) return stop('done', null, reply.output_text)
node demo.mjs
a task the agent can finish: "Where is order 1182?"
  allowance of 6000 tokens   done                    2 calls    248 tokens, 0 over
  answer: Order 1182 is out for delivery, today before 18:00.
a task it cannot: "Where is the parcel for Dana Whitfield?"
  step limit of 25 alone     stopped: step_limit    25 calls  52405 tokens
  its first call cost 58 tokens and its last cost 4135
  allowance of 6000 tokens   stopped: token_budget   9 calls   6639 tokens, 639 over
  6000 with 1500 held back   stopped: token_budget   8 calls   5222 tokens, 0 over
  partial result: 8 tool results, the last for "Whitfield Dana"
ten runs of the first task for one tenant, each with its own allowance of 6000
  per-task allowance alone   10 done, 0 stopped, 2480 tokens for the tenant
  tenant allowance of 1000   4 done, 6 stopped, 1045 tokens for the tenant

The allowance ended the runaway task after nine calls, at about an eighth of what the step limit let through. It also ran 639 tokens over. The ninth call was admitted with 5,222 spent, and the feature learns what a call costs only when the response arrives. Holding back 1,500 tokens, a little more than that call, ended the run under the allowance. Either way the caller gets the tool results gathered so far and a reason, not an exception from the middle of a step.

Add an allowance for the tenant

A per-task allowance says nothing about the sum. Ten runs for one tenant each finished far below 6,000 tokens and together spent 2,480 that no limit covered. Keep a second counter per tenant, feed it from the sink of every task’s client, and check it between calls as well.

export function tenantAllowance(name, limit) {
  let used = 0
  return {
    name,
    limit,
    add(tokens) { used += tokens },
    get used() { return used },
    get spent() { return used >= limit },
  }
}

With a tenant allowance of 1,000, the fifth run stopped after one call and the rest made none. That counter lives in one process. Tenants served by several processes need a shared store.

Keep the same count in a loop with no SDK

The counter does not need a generated client. Any SDK that returns the usage object works, whichever generator produced it: charge the usage after each call and ask before the next.

    for _ in range(max_steps):
        # Between calls: the only place a loop can stop without losing a response it paid for.
        if budget.spent:
            return stop("stopped", "token_budget")
        reply = model({"model": "quill-small", "input": items, "max_output_tokens": 256})
        calls += 1
        budget.charge(reply["usage"])

Between calls is the one place a loop can stop without throwing away a response it has paid for. The counter itself refuses a usage object it cannot read, so a changed response shape fails loudly and is never counted as zero.

    def charge(self, usage):
        """Add one response's usage. A usage object it cannot read is an error, never a zero."""
        tokens = usage.get("total_tokens") if isinstance(usage, dict) else None
        if isinstance(tokens, bool) or not isinstance(tokens, int) or tokens < 0:
            raise ValueError(f"usage carries no usable total_tokens: {usage!r}")
        self.used += tokens
        return tokens

    @property
    def spent(self):
        """True once another call may not start. Checked between calls, so it is a cutoff."""
        return self.used >= self.allowance - self.hold_back
python3 agent.py
own loop, no SDK: "Where is the parcel for Dana Whitfield?"
  allowance of 6000 tokens   stopped: token_budget   9 calls   6639 tokens, 639 over
  6000 with 1500 held back   stopped: token_budget   8 calls   5222 tokens, 0 over

Enforce the allowance inside a Speakeasy or Fern SDK

Speakeasy and Fern document no budget option. Speakeasy’s extension list has none, and its mechanism for this is an SDK hook. beforeRequest may throw to stop a request, and afterSuccess receives the response before it is deserialized, which is where the usage can be read. The hook context carries the operation id and the SDK’s options, so the allowance has to be found through those. Hooks are registered in src/hooks/registration.ts, a file generated once and never overwritten, so they belong to whoever generates the SDK. They are available to Business and Enterprise users only. If you consume a Speakeasy-generated SDK, count in the loop.

The README of Fern’s open source repository says that project is no longer maintained and points to the hosted service. Its current sdk-config.yml format has no budget key. It does have a setting that makes the generated TypeScript client accept a fetcher option, which wraps every HTTP request:

targets:
  - language: typescript
    generation:
      allowCustomFetcher: true

Fern documents the fetcher as middleware for authentication. A fetcher that refuses when the allowance is gone, and counts usage on the way back, puts the check inside the client by the same route. The SDK’s producer has to switch the setting on, and each user supplies the function.

Check it worked

Run the suite. It checks calls and tokens against the stand-in’s own counts.

node --test budget.test.mjs
1..13
# tests 13
# suites 0
# pass 13
# fail 0
# cancelled 0
# skipped 0
# todo 0

# pass 13. One test asserts that the runaway task stops with token_budget after 9 calls and 6639 tokens, and that the tenth call never reached the model. Another runs the Python loop and expects the same two numbers.

When it goes wrong

The allowance never trips. The responses carried no usage.total_tokens, so each call was priced at zero. Set unit to a worst-case call, as the client on this page does: it applies only when the body figure is missing.

An allowance of zero stops nothing. The feature reads budget: 0 as no limit, so settle that case before the first call.

A network failure is reported as a spent budget. The loop caught every error. Match the cost_budget code and throw the rest again.

The run ends over the allowance. One call can cost more than what remained. Hold back the size of your largest call, and cap its output with max_output_tokens.

When not to do this

Do not present a between-calls cutoff as a cap. One oversized call passes it, and the tokens are spent by the time the count arrives.

Do not switch model clients to sdkgen for this counter. The Python loop here is the same allowance in one small class, with any SDK. sdkgen’s budget earns its place when the model client is generated by sdkgen already.

Do not use sdkgen’s opt-in feature where the stop must not be optional. A Speakeasy hook is compiled into every copy of the SDK you publish, and sdkgen’s feature waits for each caller to activate it. Fern’s fetcher option is the better fit when each user should bring a policy of their own, because it hands them every request.

Do not rely on a step limit for cost. LangGraph’s default allows 10,007 super-steps.2

Do not skip the proxy when many processes share the spend. LiteLLM refuses a call across all of them, provided its database is connected.3

Last verified

Verified 2026-11-19 against Node 22.23.5, Python 3.13.18, @voxgig/create-sdkgen 0.31.9, and @voxgig/sdkgen 4.44.3. Every output block is what the command preceding it printed. No request reached a model provider: the model is a stand-in, and its token counts are a simplification. Speakeasy and Fern were not run; their options come from their documentation. LangGraph, LiteLLM, and the OpenAI Agents SDK were not run either.

Footnotes

  1. The features reference says the budget is per client instance, so two processes get two budgets and neither knows about the other. It stops this client, the same passage says, not your account. It also calls the cutoff the useful shape for agent traffic, on the grounds that such traffic is bursty, repetitive, and prone to tight retry loops. ↩︎ Back to text

  2. The LangGraph documentation dates the figure: 10,000 steps between versions 1.0.6 and 1.1.3, and 10,007 super-steps starting in 1.1.4. The page does not say what the seven are for. The result is a prime number. ↩︎ Back to text

  3. LiteLLM’s budgets page says every budget on it is enforced against spend read from the database, so none of them caps anything on a deployment without one. The global budget, it adds, fails open there: the check is skipped and requests keep being served past the limit. A warning is logged once at startup. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.