How-to › Move data in and out

How to paginate an API with Lua coroutines#

Wrap a cursor-paged endpoint in a coroutine.wrap iterator so callers write one for loop, and raise a failed page as a table that `pcall` can catch.

Audience
API consumer
Level
intermediate
Topic
Paginate collections
Languages
Lua
Verified

Read first: Loop over every page of a REST collection

Your Lua code calls a list endpoint that returns three orders and a next_cursor, and every caller repeats the same while loop around it. One caller forgets the cursor and reports three orders out of seven. Another treats the nil, "503", body the generated client returns as an empty page, so the failure never reaches its error handler.

What you get

You will end up with an iterator that lets a caller write for order in orders() do, fetches each page only when the loop needs it, and raises a failed page as a table pcall can catch. This is for you if you consume a paged API from Lua 5.4.

Short answer

In Lua 5.4, wrap the page loop in coroutine.wrap and yield each item. The caller writes for order in orders() do, and the next page is fetched only when the loop reaches it. Raise a failed page with error({ status = 503 }, 0) inside the producer, and pcall around the loop receives that table. When the HTTP layer yields to a scheduler, use a closure iterator instead, which keeps the cursor in upvalues.

You will need

Lua 5.4, and for live requests an HTTP library such as lua-http plus a JSON library. Verified 2026-11-20 against Lua 5.4.6, @voxgig/sdkgen 4.44.3, OpenAPI Generator 7.27.0, and dkjson 2.8. The sample runs on Lua alone. fake_api.lua stands in for the API in memory, and standin/http/ replaces the two lua-http modules the generated client loads, because lua-http needs cqueues, luaossl, and lpeg, which are C modules.1 The document, api.json, pages seven orders three at a time behind data, next_cursor, and has_more.

Approaches compared

ApproachWhen it fitsWhat it costs youWhen to pick something else
closure-based iteratorYou want no coroutine, or the HTTP layer yields to a schedulerPage state spread over several upvalues, and a refill loop that is easy to get wrongThe producer reads better as a plain loop and nothing beneath it yields
coroutine.wrap iteratorThe producer reads as a plain loop over pages, and the transport blocksAny yield from the transport reaches your for loop as if it were an itemThe HTTP layer runs under cqueues and yields while it waits
OpenAPI Generator lua clientYou want a generated client and can install lua-http with its C dependenciesA beta generator, no paging helper, and string-only arguments, so the first page needs an empty cursorYour runtime cannot load cqueues, luaossl, or lpeg
progenitorThe consumer is written in Rust and the API follows Dropshot’s paginationNo Lua output, so a Lua caller gets a design to copy and no codeAny Lua consumer
sdkgen Lua paging featureThe API owner ships an sdkgen Lua SDK and you can turn on pagingAn iterator over one call only, a control table on every call, and luasocket for live requestsThe API has no sdkgen SDK, or its SDK was generated without paging

Voxgig maintains sdkgen. It is one of five options here, not the recommendation.

The two hand-written iterators cost under twenty lines each and work over any client. OpenAPI Generator’s client needs those lines too, plus lua-http’s C modules. sdkgen’s paging feature reads the cursor for you, which removes the cursor code but still leaves the iterator to write.

Yield each order from a coroutine

coroutine.wrap returns a function, and a generic for calls it until it returns nil. The body is an ordinary loop over pages that yields one item at a time, so the second page is fetched only after the loop has taken the third order.

function M.with_coroutine(fetch_page)
  return coroutine.wrap(function()
    local cursor, seen = nil, {}
    repeat
      local items, next_cursor = fetch_page(cursor)
      for _, item in ipairs(items) do
        coroutine.yield(item)
      end
      if next_cursor ~= nil then check_cursor(seen, next_cursor) end
      cursor = next_cursor
    until cursor == nil
  end)
end

fetch_page(cursor) returns a page’s items and the next cursor, or raises. check_cursor raises when the API hands back a cursor it already gave, which would otherwise loop forever. An error inside the producer leaves through the wrapped function: the manual says it closes the coroutine and propagates the error, so a pcall around the caller’s loop receives it unchanged.

Keep the page state in a closure instead

The same sequence needs no coroutine. The cursor, the current page, and the index live in upvalues of the returned function, which refills its page when the index runs past it.2

function M.with_closure(fetch_page)
  local items, index, cursor, done, seen = {}, 0, nil, false, {}
  return function()
    while index >= #items do
      if done then return nil end
      local next_cursor
      items, next_cursor = fetch_page(cursor)
      index = 0
      if next_cursor == nil then
        done = true
      else
        check_cursor(seen, next_cursor)
        cursor = next_cursor
      end
    end
    index = index + 1
    return items[index]
  end
end

The while handles an empty page that still carries a cursor. Written as an if, that page ends the loop early, and a test catches it.

Drive the sdkgen paging feature from the same iterator

generate.sh built the clients once and committed them under generated/. The sdkgen part adds the lua target and the paging feature.

(cd "$WORK" && npx --yes @voxgig/create-sdkgen@0.31.9 orderdesk --def api.json \
  --target lua --feature paging)
(cd "$WORK/orderdesk-sdk/.sdk" && npm run generate)

The generated Lua SDK has no iterator that crosses pages. Its entity’s stream("list") yields the items of one call, three orders from one request here, and a test pins that. list(match, ctrl) returns one page of entities and an error, and the paging feature writes the server’s cursor into ctrl.paging. Passing the same ctrl again sends that cursor, so the page function ignores the argument the iterator hands it.

function M.sdkgen(client)
  local orders = client:Order()
  local ctrl = { paging = {} }
  return function()
    local entities, err = orders:list(nil, ctrl)
    if err ~= nil then
      error({ kind = "sdk", message = tostring(err) }, 0)
    end
    local items = {}
    for i, entity in ipairs(entities) do
      items[i] = entity:data_get()
    end
    local paging = ctrl.paging
    return items, paging.hasMore and (paging.cursor or paging.nextPage) or nil
  end
end

ctrl must start as { paging = {} }. With a bare {}, the feature records nothing in it, and the next call asks for the first page again, which a test pins. The SDK also returns its error rather than raising it, so the function raises a table, and error adds a position prefix only to a string message.

Page through OpenAPI Generator and progenitor clients

OpenAPI Generator’s lua generator, rated beta on its documentation page, writes one function per operation over lua-http and dkjson, leaving paging to the caller. The query string is built from both arguments every time:

function default_api:list_orders(limit, cursor)
	local req = http_request.new_from_uri({
		scheme = self.default_scheme;
		host = self.host;
		port = self.port;
		path = string.format("%s/orders?limit=%s&cursor=%s",
			self.basePath, http_util.encodeURIComponent(limit), http_util.encodeURIComponent(cursor));
	})

encodeURIComponent calls str:gsub, so limit must be a string and the first page needs cursor as ""; the stand-in reads an empty cursor as the start, so check what your API does with one. A failed request returns nil, the status as a string, and the body, and the adapter in clients.oag raises those as { kind = "http", status, body }.

progenitor has no Lua target. It writes Rust, and its documented paging mechanism is an extra _stream method for operations that use Dropshot’s pagination convention. Run on this page’s document, it wrote two methods and no stream:

lua5.4 inspect_progenitor.lua
language: Rust (lib.rs, 9313 bytes)
operations: list_orders, get_order
stream methods: none
mentions of x-dropshot-pagination in api.json: 0

A Lua consumer of a Rust-first API takes OpenAPI Generator’s client or the sdkgen SDK, and wraps it in one of the two iterators here.

Check it worked

demo.lua runs one loop three ways and prints the requests each made.

lua5.4 demo.lua
coroutine.wrap over OpenAPI Generator  7 orders: ord_101 ord_102 ord_103 ord_104 ord_105 ord_106 ord_107
    GET /v1/orders?limit=3&cursor=
    GET /v1/orders?limit=3&cursor=cur_0003
    GET /v1/orders?limit=3&cursor=cur_0006
closure over OpenAPI Generator         7 orders: ord_101 ord_102 ord_103 ord_104 ord_105 ord_106 ord_107
    GET /v1/orders?limit=3&cursor=
    GET /v1/orders?limit=3&cursor=cur_0003
    GET /v1/orders?limit=3&cursor=cur_0006
coroutine.wrap over sdkgen paging      7 orders: ord_101 ord_102 ord_103 ord_104 ord_105 ord_106 ord_107
    GET /v1/orders?page=1
    GET /v1/orders?cursor=cur_0003
    GET /v1/orders?cursor=cur_0006
break after 4 orders: 2 requests

Seven orders arrive in three requests by every route. sdkgen’s first request carries page=1, because the feature sends a page number until a response returns a cursor. The last line is the laziness: a loop that stops at the fourth order made two requests, not three. test.lua asserts each of these, and the error and scheduler cases below, and exits 1 on any failure.

lua5.4 test.lua
20 tests, 20 passed, 0 failed

When it goes wrong

A page fails and the loop carries on as if the list were shorter. The generated clients return an error instead of raising it. Raise it in the page function, as clients.lua does, and wrap the loop in pcall.

lua5.4 pitfall.lua
1. page 2 answers 503: pcall around the loop gets the error table
   with_coroutine  ok=false, orders before it: 3, error: kind=http status=503 body={"error":"try again later"}
   with_closure    ok=false, orders before it: 3, error: kind=http status=503 body={"error":"try again later"}

2. the generated list_orders called with a number and nil
   ok=false, standin/http/util.lua:16: attempt to index a number value (local 'str')

3. a transport that yields while it waits, run inside a scheduler coroutine
   with_coroutine  scheduler saw 0 waits; loop saw 10 items: <waiting for the socket> ord_101 ord_102 ord_103 <waiting for the socket> ord_104 ord_105 ord_106 <waiting for the socket> ord_107
   with_closure    scheduler saw 3 waits; loop saw 7 items: ord_101 ord_102 ord_103 ord_104 ord_105 ord_106 ord_107

The first call to list_orders raises before any request. The stand-in’s copy of lua-http’s encodeURIComponent indexed the number 3, so the real module names its own file in the message. Pass tostring(limit) and cursor or "".

The loop receives values it never asked for when the iterator runs under cqueues. lua-http promises not to block when it runs inside a cqueue, and it waits by yielding, but coroutine.yield suspends the innermost coroutine, which is the coroutine.wrap producer. The third block is a model of that, with a stand-in transport that yields a string while it waits, and not a cqueues run. Its yield went to the for loop instead of the scheduler. Use the closure iterator, which adds no coroutine between the transport and the scheduler. cqueues also ships auxlib.wrap, whose source says it passes a polling yield up from wrapped code; it was not run here.

When not to do this

Do not use the plain coroutine.wrap iterator under a scheduler that runs on coroutines, such as cqueues. The transport’s yields land in your loop. The closure iterator composes there without a library’s help.

Leave sdkgen out of a Lua consumer of someone else’s API. Its paging feature removes the cursor bookkeeping but still leaves the iterator to you, and its live transport needs luasocket. OpenAPI Generator is the better choice when the vendor publishes no sdkgen SDK and you already run lua-http.

Do not wait for progenitor to emit Lua. It generates Rust only, so it suits a Rust service in front of a Dropshot API, not a Lua consumer.

Last verified

Verified 2026-11-20 against Lua 5.4.6, Node 22.23.5, @voxgig/create-sdkgen 0.31.9, @voxgig/sdkgen 4.44.3, OpenAPI Generator 7.27.0, cargo-progenitor 0.17.0, dkjson 2.8, and basexx 0.4.1. Every output block is what the command preceding it printed. The three generators ran once, in generate.sh, and their output is committed under generated/. No request left the process: fake_api.lua answered both clients through standin/http/, a model of lua-http’s two modules, so the scheduler case is a model too. lua-http, cqueues, its auxlib.wrap, and luasocket were not run.

Footnotes

  1. The lua-http introduction lists the operating systems that cqueues and OpenSSL support, at the time of writing, as GNU/Linux, FreeBSD, NetBSD, OpenBSD, OSX, and Solaris. A pure Lua library that reaches the network through C modules runs where those modules build, which is a shorter list than the places Lua runs. ↩︎ Back to text

  2. Both techniques are older than this Lua. Programming in Lua gives closures as iterators in section 7.1 and coroutines as iterators in section 9.3. The online copy carries a banner saying it is the first edition, written for Lua 5.0. The same banner says the fourth edition targets Lua 5.3, one release short of the interpreter that ran this page. ↩︎ Back to text

Read this page as markdown · All how-to guides

Generate the client instead of writing it#

Retries, timeouts, pagination and auth are the same problems in every client. Voxgig generates them from your OpenAPI description, in 23 languages, from one model.

Listen to Fireside, the Voxgig podcast

Since 2018, Richard Rodger has talked with developer advocates, event organizers, and the people who build developer-facing companies about the specifics of the work.